 Thank you, Sue, for the invitation, and thank you to the folks in the Ford School who have hosted me today. I'm going to be talking about how schools respond to differences in teacher effectiveness. We now have at least a decade of tons and tons of research indicating that teachers are the most important in school contributors to student achievement. That's probably not a finding that your grandmother would have needed a decade of research to tell you. This idea that some teachers are better at their jobs than other teachers. But we now have measures, however imperfect, or proxies for how much teachers contribute to their students' achievement. But there's widespread concern that the rigidities of the public school system make it unresponsive to teacher quality. So even though there are these differences, and we now have can identify them based on test scores, and we probably could have identified them before based on things like principal observations, that things like collective bargaining agreements and the structure of the public school system mean that it's not going to respond to those differences even when they are apparent. So the policy, where policy is now is that a lot of districts in many states are thinking about changes to their teacher evaluation systems to take into account effectiveness. But we don't know a whole lot about how schools sort of under existing systems over the last five or ten years already respond to differences in teacher effectiveness. Are they really constrained? And when they're not constrained, do we see some responses? So I'm going to talk about three research questions today. The first is how are attrition and mobility, so leaving the teaching profession entirely and moving from one school to another, how are those things related to effectiveness among early career teachers, people who have just started out as teachers? Second, how is effectiveness related to promotions and strengths? I'm going to show you some data indicating that a lot of mobility out of teaching is actually into other jobs in public schools. So how does that relate to effectiveness in the classroom? And finally, how is effectiveness related to compensation both inside and outside of teaching? You can probably guess the answer to the first part of that about the relationship between compensation and the salaries that teachers earn in the classroom. But I'm going to show you some new evidence on the relationship between effectiveness and earnings of former teachers once they've left the classroom. So these three studies all exploit unique features. Florida's a very rich longitudinal data system, pre-K to 20 education data warehouse. It's based on three papers, all of which are joint work with Marty West from the Harvard Graduate School of Education. And there's some titles there. If you want to learn more about these papers, the details are in those papers. So I'm going to give basically a quick overview of each of these three papers and talk a little bit about how they tie together. Then I'll look forward to questions and discussion. So the research design is pretty straightforward. We measure teacher effectiveness for the purpose of these three studies as value added to math and reading scores in grades forced to rate. As anyone who knows anything about value added knows, these are not perfect measures of teacher effectiveness. They're noisy, bounce around from one year to the next. We try to mitigate that by averaging over years and over multiple tests. But there's no perfect solution to that problem. And we know there's not a random sorting of teachers and students. So these measures are limited, but for research purposes, I think they're pretty good. So we're going to look at the relationship between those estimated effectiveness measures, which is these value added to test scores. So when I say effectiveness, I'm talking about value added to test scores. And mobility across schools and out of teaching, changes in job assignments within public schools, and wages outside of teaching. Just to sort of start out with a simple descriptive picture of what the career paths of new fourth to eighth grade math and reading teachers look like in Florida over during the first decade of the 2000s. So we start out with teachers, new teachers. They are in these high stakes classrooms. These are tested grades and subjects in Florida, fourth to eighth grade. In their first year, we look at the second year. And after just one year, three quarters are still in the same school that they started out in. About 18% are in a different school, and 7% have left the public schools entirely. And then some small slivers, the purple and red there are, they're non-teachers in a different school or they're a non-teacher in the same school. So a non-teacher could be a principal, or a guidance counselor, or librarian, or bus driver, any number of things. And you go out, after the fourth year, a majority are no longer in their initial school. And you go out to say six or seven years, and 23, 28% are in the same school. So most teachers don't stay in the same school that they started out in. And in fact, a larger number move on to a different school and a substantial share leave the public schools entirely. So this seems to create some potential for responses to teacher effectiveness. Maybe the folks who are less effective in one school move to another, or maybe they exit the teaching profession entirely. But that's not the case. That's what this chart shows you. So what this shows you is the share of new teachers remaining in the same school over this seven year period by the quartile of their value added. So what this shows you is that over time, teachers leave their initial school. But it bears little relationship to their value added. So among teachers who are estimated to be in the top quarter of effectiveness, based on a value added, 26% are still around in that school after seven years. And in the bottom quarter, it's 23%, and in the middle quarters, it's about 22%. So is this good news or bad news? So if you came into this really worried that the best teachers were all going running from the classroom to do other things, to make more money and other professions, or they were going and running to other schools. Well then this is good news for you, because that's not the case. They're not leaving any more than anyone else is. And in fact, they're actually a little bit more likely to stay. But if you're thinking about, if you're the principal of a school with numbers that look like this, I think maybe you'd be a little bit disappointed, because in any industry you'd probably hope that your best performers would be more likely to stick around over time. And your worst performers would be weeded out. And the initial thing you could say, well, after three, four years, teachers get tenure, principals really don't have authority to do that. But these are new teachers. So you look between the third and fourth year, or even just the first three years here, this is a period when principals are not constrained by tenure loss. They can get rid of people without a whole lot of difficulty. Yet even when they're not constrained, you don't see any evidence of significant behavior, especially between the third and fourth year, where tenure occurs, that there's some kind of weeding out process that's related to quality. Now you might worry, well, this is the picture overall. What about schools that serve a high proportion of disadvantaged students? Those of the schools were really worried about the best teachers. They're all running off maybe not to jobs outside of public education, but to jobs in schools that serve more advantaged students, where maybe the working conditions are better, where maybe the salary structure is a little better. So if you're worried about that, this is once again sort of good news the way the other one was good news. If you compare the high SCS schools and the low SCS schools, and this is just defined in terms of the share of kids eligible for free lunch at those schools, you see that the share remaining is higher at the higher SCS schools. So the school serving disadvantaged kids are more likely to lose their teachers. It's not a huge difference, something like 10 percentage points. But at the same time, the difference between the top quarter and bottom quarter of teachers in terms of their value added isn't all that different. So it's true that people leave and the people who are more effective, are a little more likely to stay. But that pattern isn't all that different at the high SCS and low SCS schools. So there doesn't seem to be too much sort of systematically behind this fear that the best teachers rather are leaving the schools that need them the most. So now this turns from the share of new teachers remaining in the first school where they started out to the share of new teachers remaining in the Florida public schools overall. So maybe you take a look at a chart like this and you say, well, people are leaving their school, but maybe the top quartile, the best people they're leaving and going somewhere else because they got a better opportunity or something more desirable for them. And the people at the bottom, they left this school but then they didn't find any other opportunities because they couldn't get a good reference from their first school because their performance wasn't so great. This shows that's not the case. There doesn't seem to be any sort of weeding out of low performers or higher retention of high performers in the public school system in Florida as a whole. After seven years, 74% of the top quarter are still teaching somewhere in the Florida public schools as compared to 71% of the bottom quarter, 69% of the third quarter. They're pretty tightly clustered together and it suggests a story of, for the low performers of sort of a dance to the lemons where you can't, you leave one place, you can still get a job somewhere else, even if things weren't so great at the first place. So that's a picture of sort of these overall patterns of movement across schools and movement out of the profession. But a lot of that movement within the profession occurs across jobs within the public school system. So people will stop being a fourth grade math teacher, maybe they'll become a high school science teacher or a second grade classroom teacher or a principal or a reading coach. So this next study I'm gonna talk about looks at those promotion reassignment patterns. And it's motivated by this evidence we have from low-stakes surveys and from formal evaluation programs, which indicates that administrators, usually school principals and assistant principals, can reliably identify their most and least effective teachers. So they know something about teacher effectiveness, even independent of this value added information that I have as part of this study. But the concern here is that policy rigidities related to compensation and dismissal, things like tenure and the salary structure, constrained administrators ability to respond to differences in effectiveness. At least later on in their careers. But it's possible that administrators will exercise more control over job placements within school districts. So maybe they don't feel like they either can or want to get rid of someone who's not doing such a great job as a fourth grade teacher. But maybe they'll move them to an area where they'll do less harm or maybe the person genuinely feels that person will be more effective, a better match for them. At the same time, maybe the higher performers will be promoted into positions of greater responsibility. So we look at this descriptively in Florida by looking at the relationship between effectiveness for these, for a cohort of about 25,000 teachers in grades four to eight math and reading. So they're starting out in four to eight math and reading so we can measure their value added. And then we see where they go over a number of years from 2001 to 2008, 2009. And we divide their subsequent positions into six groups. So first is on the principal track, which means that they're either in assistant or an interim principal, or in some cases, principals over the short period of time. Usually they're just, they're getting started on that track. So they're assistant principals. Two, working as instructional coaches and reading and math. Three, teaching in high stakes classrooms. So that means they're either staying in the classroom they're already in, which is fourth to eighth grade math and reading. Or they're in math and reading in grades three, nine, or ten, which are also tested in Florida, and science in five, eight, and eleven. Fourth, they've moved to teaching in low stakes classroom positions. So low stakes just means a grade and subject that's not a subject to testing under the state accountability system. So if you're a seventh grade science teacher, that's for that year, it's low stakes. If you're a second grade teacher, that's low stakes in the sense that second grade isn't tested in Florida. Fifth, working in Florida school districts, but not as a teacher of record. So that includes teachers who are assigned to assist special education students, things like that, but you're not in the course records. We're finally no longer employed by Florida school districts, which I showed you before is a substantial number of teachers over time. So this just shows you what happens to this cohort of teachers from when they start in 2002 to when our data end in 2009. So about half after those seven years are still in a high stakes classroom. And then about 19% are not in the public schools. 15% have moved to low stakes classroom. So I guess that's the biggest group among those who have left the high stakes classroom but are still in the public schools. 9% not a teacher of record, 4% on the principal track, and 2% reading and math coaches. So what this does is it looks at people who move to these different positions and shows you their average value added by position. And these are presented in teacher level standard deviations. So for example, that first bar shows you is that the 830 teachers in these data, who started out in a fourth, eighth grade math and reading classroom, ended up at some point on the principal track. We're on average 13% of a teacher level standard deviation above average in terms of their effectiveness as compared to the average teacher. So principal track math and meeting coach looks similar. So what this suggests is that there's something going on here where teachers who are more effective in the classroom are more likely to be what we call promoted and that they're moving to positions of greater responsibility in terms of being a principal or a coach focusing on assisting students with instruction and reading and math. The teachers who stay in high stakes classrooms are modestly above average. And then where you really see this big negative bar, teachers who move to the low stakes classrooms. It's a pretty big group, about 7,000 teachers. So they are below average by about as much as teachers who become principals or assistant principals are above average. And the teachers who are not a teacher of record or not in the public schools are just modestly below average. So because this low stakes classroom group is so important, because these are often courses taken by students, academic courses. And that's where a bunch of teachers seem to be moving who are below average. We split that up further into different low stakes positions. And where the least effective teachers seem to be moving is to academic courses in grades pre-kated to. So these are things like first and second grade classrooms. I think this is a troubling finding because if someone is not particularly effective at teaching fourth grade, and we move them to second grade, because that's not tested, clearly the second grade students will just in one year become third grade students who are tested. So it's in the short run, it may be a strategy to improve test scores or maximize test scores. But in the long run, it's not such a great strategy. Not too much of a deviation from average in terms of the teachers who become high school academic teachers, grades 11 and 12. Science teachers, so these are low stakes science grades are modestly below average in terms of effectiveness. And it looks similar for teachers who go to other positions or to some combination of these positions. So this is all talking about movement within the public school system to different jobs. So the final research question here looks outside of teaching and looks at what happens to teachers after they leave the classroom in terms of their earnings. And how we're able to do this is we merge statewide databases on education from the Florida Education Warehouse and then from earnings records which are tracked as part of the unemployment insurance system in Florida to track former classroom teachers into new jobs. So we have a sample of 130,000 K-12 teachers. A number of whom left public schools during the period that we observed them from 2002 to 2007. And these unemployment compensation records give us the quarterly earnings of each current and former teacher working in Florida from 2001 through 2008. So we know we have a consistent measure of earnings for these teachers both while they're our teachers in the classroom, but also after they leave the classroom and either, in some cases, many of them stay in the public schools. And some of them go into other industries. And what we do is we compare the relationship between effectiveness and earnings for the same group of teachers while those teachers are in fact teachers and when they're former teachers. They've left the classroom for a job outside of public education. So what this figure shows you is the average annual earnings for this group of 9,000 teachers who eventually left the classroom. And it's their earnings while they're teaching and while they're not teaching. So it's probably not too surprising that earnings while teaching are more compressed because these teachers are all in the same industry. And we probably expected it to be more wage compression and teaching than in other industries. So the median salary of this group while they were in the classroom was about $38,000 and it fell actually a little bit to $35,000 when they left the classroom. So it wasn't the case that this group of 9,000 teachers who left the classroom were on average leaving for significantly higher paying jobs. Some of them did. If you see that the red line in that tail of the distribution there is higher than the dashed blue line, what that shows you is that there were some teachers who were able to earn more. But on average, they earned about the same or maybe a little bit less. Actually, if you split it by gender, you see I think men do a little better and women do a little worse when they leave the classroom. And you can see how much more spread out the distribution is. The ratio between the 90th and the 50th percentile is 1.3 for this group while they're in the classroom and 1.8 for the same group after they left the classroom. So now if you look at the relationship between the quarters of the value added measure that I was talking about before and earnings while in the classroom, you don't see too much of a relationship. And this is exactly what you'd expect given what we know about how teacher salary structure works. Teachers receive a salary that in most places is based on whether or not they have a master's degree and their number of years of experience as a teacher in that district or in that state. So we wouldn't expect there's a weak relationship between experience and effectiveness. We might see a small relationship, but we don't see a whole lot, right? Compared to the bottom quartile, which is left out of this chart, second and third quartile teachers make about 1% less. Top quartile teachers in terms of their effectiveness make about 3% more. So pretty small differences. Now when we turn to the same exact group of teachers after they've left the classroom, we found that those who were more effective in the classroom earn more outside of teaching than those who were less effective in the classroom when they become former teachers. So now among the same group of teachers when they become former teachers, the second quarter makes 6% more than the bottom quarter. Third quarter makes 9% more the bottom quarter. And the top quarter makes 19% more than the bottom quarter. So I think it's a pretty strong relationship and it indicates that some part of the skill that makes for an effective teacher is valued economically in the non-teaching labor market, even though there's no reward to it in terms of salary while in the classroom. So just to outline a couple conclusions, bringing these three pieces of research together indicates the school system is largely unresponsive to teacher effectiveness as many are concerned about. And what's really concerning is that even when the system is not constrained, we still see this issue. So the example I gave before was when we look at the period when principals make tenure decisions. It's a period when they're not constrained and they still make these decisions. So if you look at all the complaints out there that schools don't do enough to respond to teacher quality, on one hand they're warranted in these data support that. But at the same time it doesn't appear that enough of them are doing as much as they could do within the constraints that they have. Second, schools can respond to effectiveness in the wrong ways. This was most clear when we saw the movement of ineffective teachers from high stakes classroom positions to low stakes classroom positions. And because low stakes doesn't mean unimportant in the world, it just means not tested as part of the accountability system. Third, teacher effectiveness is related to earnings in other industries but not to earnings while teaching. So that suggests that more effective teachers have better earnings opportunities outside of teaching, higher opportunity wages than less effective teachers. So what this suggests is that, sure, there's a need to relax constraints but that alone is not going to be sufficient. Because we saw even when there weren't constraints, the behavior was maybe not what we'd like it to be. So a couple of policy implications here. I mean there's no individual policy that comes out of this as schools should do X or states should do X tomorrow. But it certainly suggests some potential strategies to make schools and districts more responsive to differences in teacher effectiveness than they currently are. So the first is to improve teacher evaluation systems and encourage their use in personnel decisions. There are efforts along those lines underway in many districts in states across the country, particularly those that promise to do so as part of the Race to the Top competition. Of course, there's concerns about the need to get those right and to do that in a responsible way. But taking advantage of this information and incorporating it into those decisions does seem to have potential for changing some of the patterns that are documented in the data from Florida. Second is to make tenure more meaningful and more flexible. So as I showed you, the issue isn't just that tenure is this very inflexible thing where once teachers get it, you can't get rid of them. You also have this problem that it's not meaningful, that it seems to be handed out more or less to anyone who sticks around for a while in many schools. Third is to continue to experiment with all alternative compensation plans to account for these differences and opportunities outside of education that some teachers have. Fourth is to end last in first out layoffs that has implications for retaining the most effective teachers and removing the least effective teachers. And finally is to improve teacher retention in high poverty schools as a strategy to improve teacher effectiveness. So don't just think about we need to reduce attrition, we need to reduce turnover, but think about what we'd want those charts to look like. We don't want low turnover where everyone stays. We want high turnover over the least effective teachers. We want them to move on and do something else. And we want low turnover, high retention among those who are the most effective. So if we could do that, that could be a strategy to improve teacher effectiveness and focusing on it in that way would be, I think, a more promising strategy than just focusing on keeping around everyone who we possibly can. So look forward to your questions and to discussion and thank you for your attention. Most striking things I thought in the graphs that you presented was the fact that to first approximation at least, if I remember the graphs right and if I interpret them right, teachers were not really responding in a very big way to the opportunity costs of staying as teachers. That is that exit from teaching wasn't dramatically higher amongst the more effective teachers, although you're- It was lower. It was lower amongst the more effective teachers even though there are opportunity costs in terms of the outside of teaching wages were higher. Which actually, I mean, that suggests that basically incentives are not gonna keep. I mean, some of the models or ways we have of thinking about how do you keep the good teachers in the public schools is you pay them better. This is saying that they're not responding to the financial incentives. Obviously it doesn't say that there's no response to this but it's saying that's not the big deal somehow. So there's something, I don't know, not complete in this picture. Right, I think it's a great question and it's hard to disentangle the response to financial rewards versus non-financial rewards. And it could be that good teachers, they also, because they're good at it, they really like doing it. So as a result, you don't need to pay them all of that opportunity wage to keep them around at the same time, it could be the case that if there was some return in the classroom, we'd see even more of them staying, right? And that there's differences in responses to both financial and non-financial incentives that we can't disentangle in these data. So I'm not sure, I'd rule out the possibility that paying more effective teachers more would have some effect on retention but I think you're right that there's a whole a number of other factors at play that are potentially important and possibly more important. But it's your reaction to people who say that the rewards and incentives should be by the building in order to increase teamwork within the building and also create an atmosphere where the teachers in the building want to get rid of the bad teachers you're creating because they're getting in the way of the bonus rather than focus on individual teacher rewards and incentives. Right, great question. So clearly to the extent you can encourage teamwork, that's a great thing. I mean, whether you design performance pay plans as group-based incentives or individual incentives is a sort of a more complicated policy question than these data can answer. And one thing to think about is sort of how large the groups are. I think that's sort of seen as the key piece here. If it's a group of two or three fifth grade teachers working together, I think you're absolutely right. There's gonna be all of a sudden if one of them isn't pulling his or her weight, they're gonna want to get rid of that person. If instead you have a school-wide plan where it's 50 different teachers, well then maybe it's hard to figure out which one person is not pulling their own weight. So I think there are trade-offs to be managed there between having incentives for individuals to perform but also incentives for teamwork and certainly not having incentives that discourage teamwork. Any difference in differential rates of attrition but then also differential, like you showed how some teachers will become coaches and others will go on principal tracks by what kind of district it is, whether it's a really large urban district in Florida or whether it's a more rural district, does that actually change the picture? That's a great question. That's not something we took a look at in these data. How much these kinds of patterns vary in different kinds of districts but it's something we should take a look at in future work. Thanks. So when you present the results that the least effective teachers are kind of put into the, or end up in the grades that are not tested, I think it kind of makes us think maybe these principals are acting deviously in some way but I was just gonna throw two potential alternatives to give the principals some credit then see if you think maybe they might be likely or see if you think they're likely. So the first people who have had more experience than me in the classroom might laugh at me about this but do you think that perhaps not that grades one and two are less important to student learning than grades four and six but do you think, is there any evidence to show that teachers might have less impact? Like there's less, if there was value added there there might be less variation in teacher effectiveness on value added in those grades. And the second might be that teachers can be good at things that increase value added scores but or don't increase value added scores but increase maybe non-cognitive skills like persistence and stuff and that might be more important in those grades. So I was wondering if you think either of those explanations could be possible? I mean anything's possible, right? I mean it's hard to tell, right? What you'd really hope is that principals are trying to maximize student achievement or student learning and one way they do that is by trying to improve the match quality between the teachers that they have and the jobs that those teachers do. So you would certainly hope that maybe oh someone's not so good at fourth grade math and reading but maybe this person is just not great with that age and really they're gonna be better with little kids in second grade. So you certainly can't rule stories like that out and some of them are pretty plausible. So I think the overall, those differences in effectiveness by those different positions, it doesn't so much say that oh the sky is falling in Florida but that these are things that I would be concerned about and would want to try and learn more about to try and tease out what kind of story is it? Is it just principals saying well I want to maximize my scores and this is the best way to do it or is it something more subtle and it's probably makes those things. I'm from charter schools. So I know for example Florida has roughly 34% of their charter schools are managed by for profit management companies. Do you find that they're more like savvy when it comes to assigning and promoting teachers or is there really no difference between a traditional public school and a charter school? Good question. The Florida data do include charter schools and at some point it's not in any of these papers. At some point I think we started to try and think about this question. You might think that charter schools that these patterns would look different, that there'd be more weeding out of the least effective teachers and more retention of the higher performing teachers. And my recollection just like I said this is unpublished work that we just did in the data was that you didn't see any big startling differences like that. But that was a while ago when charters were somewhat newer in Florida we didn't differentiate by different kinds of charters like the for profit versus the not for profit operators. So it could be that there's more there than we found but there was certainly nothing that jumped off the page the last time I took a look. So one of the things I was wondering about is that you make some of these inferences off of the earnings differences between the well they're teaching and those who leave and have an outside wage. And I think maybe comparing wage offers would be a more appropriate measure if there are some teachers who are gonna leave and just because they're not gonna work anymore like somebody might have stayed home to raise children would underestimate the difference between how much more they're gonna earn when they leave the teaching world. Right, so if we had wage offers that would certainly be really nice. Right, so we know what they made but I mean to address sort of the second part of your question about teachers who leave and raise a family or something like that. So if you look in the unadjusted earnings data which I didn't show you today you see a huge bump at low levels of earnings. So what you see is a pretty significant substantial number of people who seem to have gone into part-time work. So what we do, I mean and those results are in the paper but so what we do for our preferred results is we estimate a teacher's probability of working full-time based on their earnings and their gender and we re-weight the data by that. So we weight our results away from people who seem to have gone into part-time work or maybe worked for a part of a year and then left to raise kids or something like that. So the idea is they should reflect to the best extent we can what people who were ostensibly working full-time were making and are not overly affected by decisions to work part-time which you rightly point out wouldn't tell you their opportunity wage it would just tell you something about a decision they made. The value added of teachers of different grades and subjects and then those labor market premiums. So a high value added first grade reading teacher versus a high value added fifth grade math teacher. You might think that the opportunity costs look very different for work outside of teaching and is there a more nuanced story or did that average premium hold? Not really, I mean in terms of there's not really a more nuanced story the best we can tell. It's not the number of teachers for whom we can do value added who then leave the classroom we have earnings data on during this period of time isn't that large. So that's why I showed you that result for everyone put together. But if we split it out so we can do we can do value added for fourth to eighth grade because the testing starts in third grade so that's your baseline. So we split it out fourth and fifth grade math and reading fourth fifth grade math fourth fifth grade reading six to eight reading six to eight math and the numbers move around but obviously there's more noise in those estimates because of being a fewer number of teachers. So in some cases maybe it looks a little smaller for the reading value added but it's really, there's no really compelling story in there that those averages mask. So I have a question about the policy implications that you recommended two policy implications. One was to improve teacher compensation and the second was to end LIFO. And I think I'm just wondering if you could comment on sort of those two policy recommendations because what I see a lot in the literature is that they tend to be talked about as two separate policy recommendations but it seems to me that some of the resistance amongst teachers and amongst unions for ending LIFO is that the fear is that what will happen is that if schools are not also at the same time given more money to compensate teachers that what LIFO will lead to is just keeping the cheapest teachers essentially which are the newest teachers and so I'm wondering if you could talk about just sort of what you think about those two together and do you see them as separate or do you see them together? I think you're right that they're absolutely linked or should be linked. So if you want to think about sort of a more evidence-based salary schedule, Jake Vigdor had a nice paper about this in Education Next a couple of years back. If you instead of having the salary schedule go up every year you made it look more like what the effectiveness curve looks like which is bigger increases in the first couple of years maybe five years depending on who you ask and then more of a leveling out after that. I think that would make the incentives around LIFO much better because now you don't face these, you face trade-offs and salary that are more closely tied to effectiveness. You're paying your fifth year teacher more than your first year teacher because you know your fifth year teacher is more effective through that experience but the pay between your fifth year teacher and your 20 year teacher isn't that different. So now your incentives aren't just to keep the cheapest people around, it's to do something that's more closely tied to effectiveness. I was also curious to know if the teachers that you did look at, what were their education backgrounds in terms of those with maybe of just the bachelors or those with maybe a master's degree or even those that chose to teach and that you tested. I mean, those with the bachelors did they maybe emphasize in mathematics or did they have sort of a general education degree? I mean, was that also looked at? So for these studies we didn't look too much at like that. I have another paper that Paul Peterson and I wrote a couple years ago looking at different correlates of teacher effectiveness using some of the same data from Florida. So revisiting some findings that a lot of other folks have looked at such as that there's not a big difference between teachers with a bachelor's degree, teachers with a master's degree. We also looked at backgrounds in terms of preparation programs and found that there wasn't a big difference in value added depending on the teacher preparation program you came from in Florida. So if you went to the University of Florida or Florida A&M or Florida International University that didn't seem to be related on average to how effective you are in the classroom. I have two kind of questions. I guess one is about the concept of value added you kind of alluded to in the beginning of your presentation. There are some issues and challenges especially in the short term like I've heard that there's like quite a bit of shuffling from year to year in terms of which quartile teachers can appear. And you sort of saying that it takes maybe a long period of time to identify who's a good teacher by this metric. And I just kind of wondered first like if you feel that's actually a long term like viable metric to use in policy because if it takes six, seven years to identify who's a good teacher hasn't largely the damage been done and they're filtering out of the system naturally anyway is it actually going to do anything? And then two alternatives to value added in terms of how we evaluate teachers as students we're currently like going through course evaluations. So like as a consumer of education I feel like we can identify very quickly who's a good teacher. It could take us probably a month and we could tell you who the best professors are. If you have any insight about whether teachers are being evaluated through student feedback metrics and how effective that seems to be. Schroes are great questions. So on the first one, you're absolutely right that these measures are noisiest when you only have one year of data and they get better. The more years of data you have, the better. In the work that I talked about today we averaged over all the years of data we had Florida actually administers both a high stakes test, the F-CAT and until 2008 they administered a low stakes test called the Stanford achievement test. So we averaged across both those two to try and deal with these errors and make the measures a little more accurate. So if anything those sorts of errors should bias or findings toward it not finding some of the relationships that we do. So some of these relationships I've showed you may understate the true relationships because of that imprecision in the data. In terms of using these measures for high stakes purposes I would sort of, I think I would sort of book end that conversation. So on one hand I don't think we want to go back to 10 years ago when there really was no evaluation at all. When everyone got observed once in a while and almost 95 to 100% of teachers got a high rating, a satisfactory rating and no one really got any meaningful feedback. There was no responsiveness to teacher effectiveness. For sure we don't want to go back there. And then on the other end we certainly don't want to have a value added measure and if it's above a certain number you get a bonus and if it's below a certain number you get fired because that would be crazy because we know that these measures are just too imprecise to ever be able to do that even with seven years of data. So I think it's a matter of thinking about how to use this information in a responsible way. And I think designing the perfect teacher evaluation system is probably beyond the scope of I'm not gonna come up with in the next 30 seconds. But I think there are sort of clear tensions between flexibility and transparency. So you think about a lot of these valuation systems that are being developed today that are 50% value added and 30% principal observations and 20% expert evaluations. To me they look like a bureaucratic solution for a bureaucratic public education system. If we're not willing to trust principles to make these sorts of decisions. And these data suggests that maybe we should be careful about putting that trust in them. When they had opportunities to respond, they didn't. So if we're not gonna put that trust in them then we need to have some kind of bureaucratic check all the boxes evaluation system that's gonna produce dumb outcomes some of the time. It's gonna make mistakes. It's gonna make different kinds of mistakes than we made before though. In the past we never got rid of anyone who we shouldn't have. But we also never got rid of anyone we should have. So clearly if we implement some kind of valuation system that bears some relationship to effectiveness we are now going to make fewer of the mistakes that we made in the past which was keeping people who should have gone. But we're gonna make more of the mistakes that we never made in the past. We're gonna get rid of people by mistake. So I think there are real challenges in managing these trade-offs between something that's fair to teachers that still makes it an attractive profession and at the same time making it a profession that's more responsive to quality than it's been previously. I'm wondering how much the results might look different particularly for the very first graph with the teachers leaving the teaching market. If we were looking at people who had background in an education degree versus people with degrees in other fields, a non-traditional certification route, whatever. Because I can see a person on that margin making the decision, okay I have this education degree. I don't think I do well in the labor market. Your results show they would but they believe they wouldn't. That's a person who ends up on the principal or on the reading coach track. The person who has a degree that they believe would give them value in the outside labor market. They're the person who leaves and winds up with the increase in wages. So I don't know if you have anything on that. Yeah, it's a great question. It would be interesting, we haven't looked at it. It'd be interesting to look at it to the extent we can. The key limitation of the Florida data in this regard is that they only indicate the college attended by teachers for those that went to a public university in the state and then we'd have the major for those teachers and only since 1995. So the share of this group of, depending on how many years we need to look at, but five to 20,000 teachers that we'd have that information for is smaller. But I think it would be worth looking at for those for whom we can do it. So I had a sort of question about the earnings announced in the chapter. I was very surprised that it was so striking how different this is. So did you, or could you be, could you basically look at what industries and what jobs you're going to, particularly industry, I imagine? Can that be used to say anything about what actually we are measuring when we're measuring value added? And I'm sorry, people in other context. So I ever said value added is sort of your ability to produce test scores. But what does that mean if you go and work in business? What skill does that have? Right, right. It's a great question. I don't think it's something we can get up particularly well with these data. We do, from the UI data, have the NAICS codes. So we have a couple, I think we have a table in the paper. Looking at that, there aren't any numbers on that that really jump out at you. So most people stay in the public schools. And the results I showed you are for the people who do not stay in public education. So the kinds of categories therein are a very small number in private schools, health services, professional services. And it's just sort of a few here and there. It's not like oh they all go and become into one category. So it's hard to know what to make of that because the categories are pretty broad and they're sort of spread out across them. So it would be nice to know what it is. Is it their ability to work towards this goal of improving the test score? Is it just something, some general cognitive skill? It would be really nice if we had something like SAT scores or just some general measure of teachers' skills just to see whether it's a piece of that that's related to value added that's driving this or it's something independent of that. So I don't know. With doing this, but I would just be curious to see if you sort of split the sample into students or into teachers that went into industries that are I think in some way kind of related to education or at least the skills required of teaching versus people that went into completely unrelated sectors. If you saw the same kind of earnings impact of value added. Yes, a good idea. We should see. So what industries would, so private schools obviously would be related. Yeah, right, that's a good idea. Right, right, that's a good idea. So the sort of standard labor economics model of earnings over the work life or that when you first get out of school, employers don't know who's productive and who isn't, all you have is where they went to school. But over time you learn about people's productivity and you might think that information starts to percolate and that principles talk to each other and so forth. So I'm wondering if you see any trajectory in the sorting. If you look at where people were gonna find out or eventually good workers start, do you then see over time they're sorting into jobs that are on some measure more desirable and earnings are not gonna be the only thing just because earnings are so compressed in this sector. I mean, in the unionized sectors, your earnings are always more compressed so differentials are just mechanically gonna be smaller. One question might be what's the rank look like? Does it look like people are starting to rank by what their value added is? And the second is if you did some sort of index of the desirability of a job. I mean teachers seem to have a sense of what's a tough place to work and what's a more desirable place to work. And if we use that observed distribution we use that to sort of add on to the earnings measure, some measure of the non-financial returns to different jobs and see if we see the better teachers are ending up in those places. So within education or outside? Or both? Within education. So within education if you look at, a lot of this movement is to other schools. After by the fifth year there's more people that have moved to a different school that are in the same school but are still teachers. So if you look at the characteristics of those schools teachers are moving to schools with higher average test scores with fewer disadvantaged kids and the like. Even though the difference, the changes in salary aren't that big. Is that by the teachers? No. So it seems to, I mean, well everyone moved to the better schools. So I guess that the story is that new people start out in the worst schools, right? And then as people retire and leave the better schools with better just meaning, you know, more desirable at working conditions, you could have some kind of movement like that. So in a, this is updated but in the earlier version in the published version of this paper, we look at that by sort of, I think we do thirds of value added because it's a relatively small number of teachers. If anything it looks like there's bigger differences for the least effective teachers in terms of the changes. But it's not, the evidence isn't that compelling. The story really is that everyone is making moves like that and it's not just one group and not another. I guess I was wondering if you could say a little bit more about the magnitudes of actually maybe the next slide. I mean, you're sort of saying, I mean, your punchline was that there isn't a huge difference, but I guess I don't know what a big difference would be in this context given how much measurement error there is about with value added. I mean, is the difference between 26, you could say 26 and 22 are pretty similar or you could say there's 4 percentage point difference and 4% over 22 is 25% or something. I don't know, I guess I didn't know what to expect on what would be a big or small difference between the groups. Right, I think you're right that it is complicated by measurement error. So, I mean, the way I think about this is so a lot of some of the previous literature wouldn't show you, wouldn't look at this. They would say the difference is 4 percentage points and oh look, we thought that the top people were gonna be leaving in droves but they're actually more likely to stay. Great, so if there weren't measurement error and I was the principal of the school or I was, this is for the state, I was the state superintendent in Florida, I think I would wanna see big differences, right? I would wanna see the top teacher staying at high rates and the bottom teachers staying at much lower rates. How much I should expect to sort of compress what I would like based on measurement error and value added, I think is a sort of harder adjustment to make. But I mean, my gut is more than this but it's not based on much. So, a lot of your policy implications and a lot of what you presented really focuses on the top and the bottom. What does your research mean for the average teacher, the teacher in the middle that's not at risk of leaving or staying or being in a high stakes or low stakes classroom or grade? That's a great question. So, I've sort of talked about teacher quality as this immutable thing and of course that's wrong, right? So, it's an oversimplification. And clearly you want people have opportunities to get better and I think one of the real problems of some of these new evaluation systems is providing targeted feedback and providing opportunities to get better. There's a nice paper by John Tyler and a colleague looking at the implementation of a evaluation system in I think Cincinnati and they find that teachers who get evaluated under this, not just you give them a value added score but they're observed by these master teachers and they get targeted feedback that it actually has significant impacts on the quality of their own teaching. So, that's a piece that's really been left out of this. I think by focusing on the top and bottom you can kind of say, well, the top people are already good so we definitely want to hold on to them and the bottom people, quartile, maybe include some teachers who just need some help and some support but probably also include some people who just aren't cut out for this. So, we want to see differences there but I think you're absolutely right about the need to think not just about hiring and firing but about training and giving folks opportunities for improvement. Ever broke the norm of no double dipping, so. This is, another thing I found interesting in your talk was that you emphasized some of this behavior or what these patterns are not behavior that are constrained and that strikes me as given my personal experience, not my research experience as being valid in the sense that, and this is at the end of one but it is supporting your results so you might take it. The, by far the worst teacher that two of my children ever had was a teacher in a selective private school which was, we were paying more than taxpayer money to send them to. I've heard about this for the rest of my life, I mean, so far, but the principles of that school even though this individual actually endangered my children in things that she did was, had been retained by the school and I can't believe that I was the first parent whoever was over the top about her and it wasn't just that she endangered them but she misunderstood acceleration and she was a science teacher and she patronized me when I tried to bring this up to her saying that if only I understood calculus she could really explain to me why she was right. This was just kind of, as you can tell, 10 years later I'm still totally infuriated but the suggestion is that it's not, and I guess the flip side again, this is a very different circumstance, there was no union protecting her, some other forces were. The flip side is that best school experienced by same two children had was in a public school where the teachers were all unionized so it does suggest that, I mean, to me, it suggests there are lots of forces at work that go beyond the literal. The show that Gephardt was all unionized. Yes, yes, and I think that's another message here that I'm not trying to say there's no effect there at unions but I think sometimes at least the public rhetoric suggests that it's there to culprits and if we only would have removed them that wouldn't be true and that doesn't correspond to my anecdotal experience and it doesn't correspond to your quantitative results. You can't have it. We can't require it. Nice, nice, jeez, all right. So in that blank, oh, so here I wonder if, so education is one of the few places, if not the only place as far as I can tell where we have these fine measures of productivity and so we can show these shocking pictures. Has anyone done this in any other industry? Athletics, athletics. Athletics, great. Okay. How about something else that matters for my social welfare? The military. The military, okay, so and what's the evidence there? And in the military? And I wonder if this is possible in healthcare where we actually are developing some fairly good measures of quality across doctors, HMOs and so forth. But I just wonder if teaching is getting beat up on because the researchers have done such a good job of pulling the data together and that a lot of this stuff, for example, is being attributed to the particular structure of employment and education but perhaps it's true in all sorts of employment. The Peter Principle was written 40 years ago and it wasn't about education. And so I would love to see people applying these tools to other industries so we can start to get some sense of whether it is about the institutional structure of teaching or perhaps using, could we do cross-state analyses of this sort in which there are institutional differences across states in some of the constraints that we're hypothesizing. I don't know, in Florida there are places that are where unions are stronger than other places. Tenure laws differ. Do we see that this sort of stuff varies across contexts? Well, I guess one important point of context for Florida during this period is during most of the period covered by these data, Florida was implementing a statewide class size reduction policy, so they were hiring people. So there may have been constraints on their ability to get rid of people simply because they needed more embodies and just couldn't afford to pay attention to stuff like this. But at the same time, I think some similar findings have been found in the Texas data, so it's not like these findings are so much of an outlier.