 My name is Kevin Cary, I'm the Director of the Education Policy Program here at the New America Foundation. And on behalf of the entire foundation, let me say welcome and thank you so much for joining us this morning for what I know is going to be a fantastic discussion about really one of the central challenges to education policy making today. As you all know there's been an intense interest in recent years about the question of teachers and their effectiveness in the classroom, their ability to help students learn. That interest is born out of a lot of econometric research suggesting both the tremendous importance of teachers in helping student learning but also the tremendous variance amongst teachers. And I think it's really that sense of variance that has driven a lot of these research and policy questions. We want to know which of our teachers are succeeding the most, are succeeding the least, how much distance there is between them and what we should do to act on that information once we have it. A lot of the debates around teacher evaluation can be, I think, categorized in one of two things. Partly we think a lot about how we should evaluate teachers and a lot of that debate is centered around standardized test scores which have this kind of magnetic attraction in the debate in pre-K through 12 education because of their controversy, because of the way they're integrated into our accountability systems. And then the other part of the debate has been around what should we do with information about teacher performance, should we use it for selection, should we use it for professional development, should we use it for pre-service training, some combination of all of the above, and how do we find out more about teacher performance outside of the realm of test scores. But there's really been, I think, an inattention in this whole debate to the many, many parts of our public education system that sit outside of this central debate and controversy over accountability and particularly in the early greats. And this is despite the fact that the evidence showing the importance of teacher quality is as strong or stronger for young children than it is for anyone. And so it's one of these things where, because we're really not quite sure what to do in earlier grades when we don't have standardized tests just kind of spinning out scores that we can run through algorithms, people have, you know, not thought about it as much as perhaps they ought to. And that's really the reason that Laura Bornfreund, our senior policy analyst, decided to tackle this very big and very important issue and really find out what's going on in the states that have for a variety of either legislative or policy reasons decided to go ahead with this process of evaluating teachers in all grades, including the earliest grades. And what she found, I think, is fascinating, fascinating sometimes and in really interesting ways, fascinating sometimes in, oh, my God, I can't believe they're doing that kind of ways. But really speaks to, I think, again, what is one of the central policy challenges in education going forward. We had a little confusion when the first report came out. We had some people emailing us wanting to know Laura's opinions about the degradation of the great coral reef outside of Australia due to the tidal, oceans of unknown. So we had to clear that up. This is, and if you have come here with a background in oceanography, I encourage you to stay. You'll learn something. It's always good to kind of think outside your field. And if you're here to learn more about teacher evaluation in the early grades, then I think you're in for a treat. So with that, Laura Boyd-Freude. Good morning, and thank you, Kevin. And that's actually true. I did receive questions about oceanography, so very, very true. My recent paper, An Ocean of Unknowns, explores how states and school districts are or are planning to use student achievement data to evaluate pre-K and early grade teachers. I looked at five states and three school districts, Colorado, Delaware, Florida, Rhode Island, and Tennessee, and Austin, Texas, Hillsborough County, Florida, and Washington, DC. Today, I'll briefly discuss what has led to the overhaul of teacher evaluation systems and highlight what makes the use of achievement data in the early grade so complex. Then I'll briefly explain the approaches being used and delve into the opportunities and risks in one of those approaches. And finally, I'll share three big recommendations that I make in the paper, as well as to highlight some key considerations I discuss. So let's get started with just the big picture. No Child Left Behind gave us the highly qualified teacher provision. And some critics really said that the focus should not be on teacher credentials of degrees and passing scores and certification exams, as was outlined in the highly qualified teacher provision, but instead, the focus should be on teacher effectiveness as measured in part by student learning outcomes. And in 2009, the new teacher project released the WIDJA report, which really helped shine a light on the poor state of teacher evaluation, finding that these systems provide very little information to help differentiate teachers for one or to help teachers improve their practice or instruction. Later that year, the Obama administration announced its Race to the Top grant program, requiring interested states to reform their teacher evaluation systems to incorporate multiple measures of teacher performance. And multiple measures could incorporate a lot of things, but they should include observations of actual teaching. And in a previous paper, we actually talk about that and stress that in the paper was watching teachers work. And one of the most controversial multiple measures that is being required is this focus on student growth. And by the way, at the time, there was $4 billion available during a time when states really needed the money, so this became a real focus. And it definitely should be a focus, but it sort of forced states to kind of think about it pretty quickly. As of 2012, more than 20 states have passed new teacher evaluation laws, at least, and even more have made some changes through regulation. And then reinforcing the focus on teacher evaluation reform and this focus on student growth in all grades, the administration made it part of flexibility waivers from some of the most unpopular and unrealistic provisions of the no child left behind. So most states now have been granted a waiver, and just about every state is really grappling with this issue. Measuring student learning in the untested grades and subjects, and those can be just quick references, could be arts, music, science in high school, but I'm focusing really on the pre-K through third grades, which I think haven't gotten as much attention. And we know that third grade is part of state testing programs, but when students enter third grade, there's not a previous year of information, so third graders are often considered untested as well. So let's take a closer look at what makes pre-K through third grades a little bit more complex. First, children's development is directly linked to their academic growth. Teachers are crucial for helping to lay the foundation for children's literacy and numeracy, as well as general knowledge and social emotional skills and executive function, which refers to things like working with others, taking turns, persisting on a challenging task. Things that we know are really important to students' success throughout their schooling and, of course, in life. So measures of children's growth used to judge a teacher's impact on learning should be based more on than just reading and math. Measures need to look at other domains of learning. Second, it's more difficult to obtain reliable and valid assessment data from young children. One time, paper and pencil tests, with or without the bubble sheets, won't work. Other attention spans are short, and sometimes a skill they were proud of, to show off one day, they'll withhold the next. Third, some assessments often look at very basic skills and we don't want teachers to narrow student learning to simply a limited range of basic skills like, say, letter identification in pre-K or kindergarten, which some of the assessments being used are just looking at those narrow skills, which, of course, are important, but they're not providing that big picture. And then fourth, different delivery models and dosage in pre-K and kindergarten. This pre-K delivery really varies state to state. Some programs are half day and some are full. This is true in kindergarten also. Some pre-K teachers are required to have bachelor's degrees and licenses, others are not. Ratios are typically lower in pre-K and often kindergarten classrooms too. And some programs use co-teachers. So states and districts really need to think through what this means and how to account for some of these, all of these variations. So there are primarily three approaches that I found being used right now across states to measure student learning growth for the purposes of teacher evaluation and these are specifically, they're being used in some of the other untested grades and subject areas, but specifically for the pre-K through third grades. First is student learning objectives. And if you haven't heard about SLOs, this is, there's three components. First, a measurable goal or objective for a teacher's students. A growth target that is set with baseline data from on student performance in mind. So teachers are thinking about where their students are in when they enter the school year. And then an assessment or another tool to measure the student's progress toward the objective. And this seems to be the method that is really taking hold across the states. And there are districts in at least 20 states that are implementing SLOs right now. And there's some examples of states up there, Colorado, Connecticut, Delaware, Georgia, Indiana, New Hampshire, and a few others. And then the second method is new and shared assessments. And these could be assessments already being used in the classroom for some students. So they're not part of the regular state standardized testing program, but they could be assessments that the district is requiring or the state is requiring on another level for teachers to already be using. But they're used for another purpose, either diagnostic or really to be used for instruction, they're formative kinds of assessments. Or these could be brand new assessments that are created by the district or the state. And then the third is this idea of shared attribution, which is really attributing the learning of one set of students to another teacher. For example, basing a kindergarten teacher's growth rating on third grade reading scores. Okay, so looking closer at student learning objectives. See several opportunities with SLOs, which I'm gonna talk about, and then we'll look at the other side of SLOs. Largely, SLOs are teacher driven and can foster collaboration and shared priorities. In Austin, for example, teachers have to write an SLO for their individual classroom, as well as work on a team SLO, so with their grade level team. First grade teachers, for example, come together to discuss appropriate objectives for their students, as well as a way to measure the progress toward that objective. And they work together to develop the target as well. SLOs also have the potential for improving instruction and for helping teachers to better meet students' individual needs, because I think the process alone really could make it easier for teachers to understand where their students are at the beginning of the year, setting that goal of where they should be and thinking about how to get them there. And good teachers will tailor their instruction appropriately and check in throughout the year to make sure things are going according to plan. Also, SLOs really could be the best of supporting a more well-rounded curriculum as opposed to narrowing it simply because it's not limited to what's being tested requiring under a specific testing program. It opens the opportunities for teachers to think more broadly and principles to develop more broad priorities around what those objectives should be for students. In some places, teachers could come up with four, five, maybe even more SLOs. It depends on district. Some places are just requiring one or two, so those could be narrower, but I think there's some opportunity there for a more well-rounded look at what students are doing as well as teachers' impact on their learning. And then finally, I think there's the likelihood that these are more supported by teachers because the data, one reason is, the data could be more immediately relevant to their lesson planning and instruction throughout the year. So I think there are many benefits of SLOs as a way to help teachers think more deeply about what their students know and are able to do throughout the year and as I said, how to get them there. I think at the same time, there's some risks that we can't overlook. First, SLOs are resource-intensive and require new or expanded expertise at both the district and school level. To do SLOs well, school districts need to set up a review process of some sort. And in Austin and D.C., this meant dedicated staff, dedicated district staff and in some cases, in other places, school staff as well. Principles and teachers also need professional development on what a high-quality SLO looks like, how to craft those good objectives, rigorous, but attainable targets, and how to select or create an appropriate assessment rubric or other measure to really see if the progress is being made. Also, there's an inability or little way to compare teachers. So using SLOs, teachers are using, excuse me, teachers using SLOs are developing different objectives and different assessments within a school and across the district. So there's nothing, you know, it's not a standardized in the same way that a test is. So that creates some challenges when you think about how do we compare what teachers are doing with students and what kind of progress is being made. And finally, well, I'll just give a quick example. Austin has taken some steps to mitigate, you know, this issue by reviewing every single SLO and matching it to a quality standard. And DC also reviews SLOs for feasibility, which helps to ensure that, you know, the targets that are being set are both, you know, rigorous and realistic. And then in Rhode Island, the state says that it's best practice for teams of teachers to come up with, you know, the same objectives and select the same assessments, but it's not required. It's, you know, that decision is ultimately left up to the districts. And then finally, I think, you know, there's a concern for potential manipulation. And, you know, the question I ask is, is it really appropriate for teachers to play such a significant role in, you know, this piece of the evaluation process when they know that the results will be tied to high stakes consequences? You know, I think it can go either way. You know, teachers may also, may be prone to set, you know, more complicated or even, you know, more complex targets for their students thinking that they can, you know, progress even more than is actually possible. So there's concerns on both sides there. So I think that, you know, one way that states are thinking about this issue with SLOs and maybe a way to improve things is to bring some uniformity to the process. And, you know, so there's a couple of places that are doing this. Colorado, for one, has an online assessment bank and it's available to districts, schools, and teachers. And the assessments in the bank have been reviewed by testing experts and educators for quality using a rubric that that same rubric is also available to districts that want to, you know, find or create their own assessments so they can use and measure them against that rubric. Delaware Department of Education brought educators together to create a common set of objectives for each grade level and subject area and then classroom teachers when they go to write their SLO for the year they select from those objectives and Delaware also has an assessment bank. And Washington, D.C. is considering an assessment bank as well and Georgia was going with an approach where districts establish the objectives and identify the assessments to be used at specific grade levels. So finally, three recommendations, big recommendations that I make are so account for specific attributes of the pre-K through third grade teachers. What's gonna work for a seventh grade science teacher isn't necessarily going to work for a first or second grade teacher. And I think that that's an important thing that policymakers need to remember. Take the time to pilot and evaluate before adding stakes, make sure that the significant kinks of these systems have been worked out and then use caution with shared attribution but I say use caution, but really by this I mean do not use shared attribution from later grades as the sole measure of student growth to evaluate early grade teachers because it doesn't really provide any useful information about their teaching for how they're working with the students that are in their classroom but it also doesn't help them with any information to improve their instruction. So we don't know anything about what's actually happening on the growth side. The observation is a different story. And then some other key considerations. And I just want to make the point again that states and districts, that assessments and their purpose matter. So I said a little bit earlier that some places are selecting assessments that are already in use, but not necessarily, but they're not for the purpose of teacher evaluation or to measure teacher effectiveness. So states and districts should really be consulting testing experts to make sure that assessments that are being used are valid and reliable for the multiple purposes for which they're being used. Also capacity. Do states and school districts really have the resources right now and expertise to do SLO as well? And if not, what's needed to get them there? And then pre-K and kindergarten. Oh, I skipped teacher and principal preparation. So I think there's a lot of room where for growth in teacher and principal preparation programs around how to use data better, how to teach teachers to look at how students are doing in previous years or even throughout the year in the classroom and using that information on a regular basis to change their instruction to make decisions about how to differentiate for different kids and be able to get them to where they need to be that year's growth of learning through over the course of the school year. And the same is true for principals. So as I mentioned earlier, principals are gonna be one level of approval process for the student learning objectives. So do they have right now what's needed to be able to approve every single SLO to identify what's a rigorous objective to know that, yes, that target is right, that 80% of your students can move up three or four levels depending on what assessment is being used. So and then if it's a teacher-created assessment, are they able to look at that assessment and know whether it's actually a high-quality assessment or not and what makes a high-quality assessment? So I think that those are some areas we need to focus on. Again, pre-K and kindergarten teachers, you know, there are special circumstances that need to be looked at in those classrooms. The teacher's role in SLOs, as I said before, you know, teachers are highly involved. So we need to think about, I think about the teacher's role, how to check to make sure that the process is fair and any opportunity for manipulation is intentional or unintentional, is low. And then finally, this issue of research, I think we always hear this, but there really is a lack of research about whether any of these approaches are valid measures of a teacher's impact on student learning, especially from early-grade classrooms. There has been some research on the use of SLOs in Austin, Denver and Charlotte in tested subjects, finding and links were found between the quality of the growth goal that was set and student achievement as measured by state tests. But we still have a lot more that we need to know. So with that, thank you very much. Thank you very much, Laura. Hello, everybody. I am Lisa Guernsey. I'm the director of the Early Education Initiative here at the New America Foundation. It's great to see everyone. It's great to see a full room here today on a summer morning. We are going to move now into a discussion. There's a lot to kind of unpack both within the report that Laura's put together, as well as just within the small kind of sample that you received here this morning in Laura's presentation. So I'm going to moderate our discussion among this really fantastic panel that Laura has coordinated here today. I'm just going to take a minute or so to introduce everybody, and then we'll launch into a couple of questions and we should make this as conversational as possible. We've got a good amount of time. It's just about 10.30 now and we just need to clear the room by about 11.30. So we've got some good time here for so much discussion and I really am looking forward to opening it up to the room as well. So you've heard from Laura, our senior policy analyst here. To her left is Kate McMahon, who is the director of the Impact, which is DCPS's the District of Columbia Public Schools Evaluation System for all school-based employees. Of course, most folks are focused on the teacher part of that, but it will be interesting to see if we want to go broader than our discussion, particularly as we talk about principles. To Kate's left is Sandy Jacobs, who is vice president and managing director for state policy at the National Council on Teacher Quality. And we're really happy to have Sandy with us here today. And then to her left, Andrew Krugley, who is all the way from Chicago here this morning with us. He's the director of education at the Educator Learning Network and the Ounceau Prevention Fund and has also a lot of experience as a principal of elementary schools. So it's gonna be great to have that perspective. And then to my right, Tom Schultz, who is project director for early childhood initiatives at the Council of Chief State School Officers here in Washington, DC, and has been a wonderful colleague to us here in the Early Education Initiative over several years and helping us work through a lot of the big policy questions that are coming up for these pre-K through third grade years. So let's get started. What I was thinking that we should do is first I wanna just maybe reiterate a couple of things that Laura said and then maybe jump into some of those questions around those SOLs, those student learning objectives. But first just does a kind of a quick reminder so what Laura had laid out in her paper is that states are taking this idea of using student achievement data and they're taking kind of three different approaches with it. And one is using this SLO approach and that they're kind of, it's very customized at a level of the individual teacher in many cases or at the individual school or a team of teachers level where those teachers are deciding what kind of assessment they would use and then how they would track growth and what kind of benchmarks they would look at. I just do wanna point out the other two approaches that she mentioned at the beginning of her presentation. One is just the idea that states are saying, oh, if we don't have testing data for our younger kids, let's make tests for our younger kids. Let's find a way to standardize that information so that we can make those comparisons that we need across teachers and across districts and across schools. And then the third way is that shared attribution idea and I just wanna kind of put a fine point on that that what Laura had been finding is that there were some places where because there wasn't enough testing data in those younger grades, they're like, well, we at least have testing data for those third, fourth and fifth graders. Let's just use some composite of that and then that kindergarten teacher, you can kind of, you can just be judged on how those third, fourth and fifth grade teachers are doing and Laura's recommendation is, that doesn't really help those kindergarten teachers very much. That's not really helping those first grade teachers improve very much. That's not really telling us anything about those kids that that teacher actually had. And so that's where that recommendation's coming from. But I think that's also open for discussion and we should put that on the table here today. So going back to the student learning objective SLO approach at first of the three, I'm very curious among all of the panels here and so why not we'll just kind of go down at least for this first question, just go down the row, what your take is on that. It does seem like the use of student learning objectives is gaining traction, Laura found it, that that was primarily what she was coming across when she was doing her research. Do you agree? Is this where we're headed, especially for these younger grades? And what are the challenges that you see with this method? So, Kate. Sure. Well, I certainly think that this is where we're headed, one of the places that we're headed. I can say that in DC public schools for the first three years of our evaluation system, our teachers in tested grades, so those who were receiving value added scores did not have SLOs. In DC we call it teacher assessed student achievement data or TAS. They didn't have that as a part of their evaluation. Their student achievement measure was simply value added. And we actually heard from teachers in those tested grades that they wanted TAS and they wanted this SLO process. And so this past year, for the first year, we changed the evaluation system for our teachers with value added data so that 35% of their evaluation is based on value added and 15% is based on TAS or the SLO process. It's also the case that in the first three years of impact the TAS component was 10% for our teachers who were in the non-tested grades and subjects. This past year we actually moved that up to 15%. So it's increased a bit for everyone and now indeed everyone has this as a component. So it's certainly a place where DCPS is moving. And I think now what we're really trying to do is to make sure that it is a rigorous measure of teacher practice. And so a lot of the challenges that Laura outlined certainly resonate and there are things that we are grappling with right now, but it's where we want to grapple. All right, Sandy. Yeah, I also agree that this is likely where we're headed. I think we're gonna see more assessments come online, hopefully because they serve a student purpose and are also useful for measuring teacher performance, especially as we move into the park and smarter balanced assessments. I think we're gonna see new things come on the table but I think there are always going to be grades and subject areas that are best measured through SLOs. And I really agree with what Kate said about that you don't want to just, for some teachers you really want it, for all teachers in Rhode Island, it's part of the system for every teacher, even if a teacher has standardized test data and that's going to be part of their evaluation, they also set SLOs. I think both in terms of the training that's required, making sure people are really able to dig in and write meaningful objectives and know how to do that. The whole culture shift that's associated with shifting to these kinds of performance systems, it's really hard to say, well you're gonna learn how to do these and set these objectives and track your students this way but you don't need to because we have another measure to weigh to measure your performance. It's just a weird message and I think there are lots of places out there though that are only using SLOs for a certain group of teachers and they're sort of this sense that it's, well it's all we have for you. So that's how we're gonna track your performance and I think that's a little worrisome as we move forward. I think the biggest challenge in actually implementing and using SLOs is that it's not easy. Saying to a teacher, so for your performance evaluation this year you need to sit down and write some goals and then maybe pull out some data from your previous performance and here's the form and fill it out and all will be well. It's of course not that simple and I think even the places that know it's not that simple have found it to be even more challenging than they thought just in terms of making sure that teachers have the vocabulary to talk about how they're tracking and measuring student performance and I think there are some teachers who don't just lack the vocabulary, they really lack the knowledge of how they're assessing and tracking their skills so there's a whole skill building part of it as well making sure that administrators have the knowledge and skill also to help teachers with that part and to know when a teacher has set an objective that either isn't measurable or isn't especially meaningful or isn't really tracking things that matter so there's these conceptual challenges and making sure we're structuring this correctly and then there's the implementation challenges of doing it well. Andrew, from the point of view of someone who's been running elementary schools what do you think about the SLO model? I have lots of thoughts. I definitely think it's here to stay and I definitely think that we need to hold teachers accountable for student achievement. Laura makes a really important point on the third page of her paper that most of our teachers are rated as excellent yet only 75% of our children are graduating high school. Our superintendent would say to us 98% of our teachers are rated excellent yet only 50% of our students are passing our state achievement test and so what are we gonna do to change that? So it is a very important measure. I have some fears and I have some concerns and they echo Laura's. One is that we start looking that we're gonna look at it too narrowly and that everything is gonna become a high stakes test and that every assessment is gonna become a high stakes assessment because it's gonna be, it's gonna all go back to performance evaluations, personnel decisions and compensation. And I think that there's a lot of fear on teachers' parts, which then gets into one of the points that Laura made about manipulation. Are they rigorous? Are administrators really following what those are or not because you don't wanna get into drama at your school? I think it's very important that we also look at other measures in terms of teacher practice, not just SLOs and then my fear is what happens to the early childhood, what happens to pre-K, what happens even before pre-K in terms of how are we assessing our very youngest children where we know that it is there that if we make a difference with our very youngest children that we're gonna make a difference as those children move forward. And how fairly are we going to be able to assess them, be able to assess the teachers in the early childhood field pre-kindergarten and kindergarten? If you chose to fish there, great, okay, Tom. All right, so I agree with my colleagues. I think this is the direction we're moving and I think building someone on Andrew's comments, I think the strengths and concerns around SLOs depend in many respects how the information's gonna be used. So if the main purpose of your teacher evaluation system would be to do individual coaching and mentoring of teachers, I would think having teachers develop SLOs and gather feedback on tailored objectives for their class of students might be a pretty good fit. If you're trying to find out whether all the teachers in your school district are implementing a set of research-based teaching practices and you're looking at issues of uniformity, I think maybe the SLO method isn't quite as good. Third and finally, I think the greatest concern is if you're using the teacher evaluation results for personnel decisions, which in fact is one of the stipulations the Department of Education has required in their negotiation of waivers to no child left behind. So in fact, this isn't kind of speculation that oh, it might be used for personnel elevations. This is a requirement. So as Laura pointed out, one of the challenges with SLOs is the challenge of comparability of teachers and uniformity. So I think it gives me great concern that SLOs would be the primary method if those data are gonna be used for hiring or firing decisions, tenure decisions, compensation decisions. I think that might be a mismatch that will be problematic down the road. So as many of you know who followed my talks here at New America for a while, I have two little kids, elementary school, actually one of them's moving on later, but I understand the elementary school system through the eyes of a parent a lot of the times these days and thinking about teachers making decisions, making decisions, having to kind of remember one fine time, fine time for this, but making some sort of judgment call about the assessment that they should use and then measuring and tracking those kids over time and kind of figuring out what those benchmarks should be and determining what's gonna be a fair benchmark or what's gonna be a quote easy benchmark that I'll just know all my kids will get to so I can just make sure that we all have a good SLO number. Those are all things that certainly came up in my mind as well as Laura was doing the research on this paper. And so what is it going to take, do you think, to if SLOs are where we're going, what is it gonna take to get us past a lot of these challenges that you all have laid out, get us into some of these questions that I just raised as well about the kinds of training that teachers might need or the kind of time that they might need to understand what they're doing. What is it gonna take a school district to get there or even a principal and teacher to get there? I'll go back to you. Sure. I would say it really does come down to helping principals and teachers know how to select good assessments and how to set good goals. Both of those things are really hard. To be entirely honest, I think harder than we had thought they were going to be when we rolled out tasks. We hadn't realized that writing a goal that will simply produce one score at the end of the year, let alone worry about the rigor, was going to be so challenging. In our review of 10,000 goals in our first year, half of them would not have necessarily produced one simple score at the end of it. That's just workability. And so I think it is a lot about giving principals and teachers the tools. So I mean, first of all, we need assessments. But then also how to select the appropriate assessment and how to set a good goal based on real baseline data. That's a hard process and it's really time consuming. And we ask principals and teachers to do so much already. And so figuring out how to really have this be a natural part of their work that is actually enhancing their teaching and is not just in order to comply with an evaluation system, but really is enhancing their instruction. That training and that preparation I think is key. Yeah, I would just echo that entirely. I think the more that we show, there's an instructional purpose here. It's gonna make you a stronger instructor because you're tracking what your students are doing because you have a clear goal and benchmarks along the way. And the more that the training is integrated into that and that we use data because it's gonna help us serve students well and not because here's a two hour workshop in the pre-service week before school starts because you have to learn how to write objectives for your new teacher evaluation system. That's what we want to try to avoid and what I fear is what a lot of places are doing right now. I think thinking, well, we make this mistake all over the place, right? Thinking that a two hour workshop, oh, now you're good to go. Without lots of follow up and actual direct coaching and let's look at the data together. It seems like especially in the early going, the more that we can have both from the school level and then at the grade level, some at least common discussions, if not common goal setting, that the fifth grade focus this year is math and within math these specific target areas and maybe your students look very different than mine and so our actual goals are going to be different but we know this is what our goal is gonna be connected to and we know what the measure is going to be but it's gonna be a real process and I think a lot of states mostly having to do with their race to the top grants and waivers but not entirely because of those things have set kind of crazy implementation timelines to go from zero to 60 in a couple of months is really hard and without giving teachers the time to really figure out how to do it well and maybe you mitigate against that by saying, all right, well, look, we are fully implementing this year because we are and that's not really negotiable but maybe the stakes aren't attached now until we can really figure things out but part of that has to be that there's a process in place that helps teachers learn from, well, maybe we didn't set it quite right this year, maybe there was a mismatch in the assessments and what we were trying to measure, maybe in fact we couldn't measure it at all but that there's a way they're getting that feedback so that they're gonna know how to do it better next time. If all we find out from our pilots is, ugh, well, everyone kind of messed this up but there isn't a plan of how to address that and move forward then all we're doing is checking our box off that, yes, we had a pilot. I think that we need to teach everyone how to understand the data and how to use the data and how to blend the data because there's so much of it. We go on data overload and then we don't know what to do with it and then we just kind of back away from it so for instance, the school in which I worked and had this opportunity the last year that I was there and things have changed dramatically since I left but we had one target for the whole school which was every single teacher is gonna have one more student at or above grade level at the end of the year than we did at the beginning of the year. Now that seems like a very easy thing to measure and that seems like something that makes total sense. One more student in the course of three years we should be able to eliminate all the children that are below grade level. But then we had to define what is grade level. We had to define what does that mean for different teachers where there weren't necessarily assessments, art, music, PE, the librarian for instance. We had to decide what were the right assessments in each of those places. So we had to decide is it gonna be a target or is it gonna be a growth model? Are we gonna say we're all gonna hit a target? So I think one of the issues that we have to look at is laying all of this stuff out before we start because on paper it made sense at the beginning and like I said, things have changed dramatically since I've left there and I know that they've made many, many tweaks since I've come to the ounce. But we have to think forward. We have to really be careful before we put something in place when there's so many questions that are still unanswered. For instance, one of the side effects was no one was willing to take a student teacher because how is that going to affect my outcome at the end of the year? Well, that's a huge implication for what we're trying to do for our field in terms of improve practice and make better teachers if you're afraid to take on a novice teacher in your classroom and let that person teach because of what is gonna happen with your SLO whether it be the same SLO for the whole school or an individual one, that's something we didn't think about. And so all of these things need to be laid out really clearly and really thought through before we start. And I think what everyone is doing is jumping in and moving forward and it's scaring people. It's scaring principals, it's scaring and it's scaring teachers and ultimately that's gonna affect kids. I can just jump in here then. So I'm curious if you're all in agreement with one of the recommendations Laura had put out which was to pilot and evaluate first. Is that something that we could all say is a good idea, good recommendation or they are tweaks on that and I don't mean to take your time away from this Tom but I thought I would just jump in with that question maybe right now but yeah, take a shot at that Tom and then anything else you were gonna add. So I would say yes, continually evaluate these systems as they're rolling out. I guess I would add on the question of what the districts need to do it. I think the directions that Laura outlined of moving towards uniformity and creating resources so that what is meant by an SLO is not that each individual teacher in America decides and negotiates with their principal somehow. I don't know how this would work. This is my SLO. The other thing, it's a larger point is that I think we've gotta do what we can to line up the incentives and the messages from this teacher evaluation initiative with the other things that we say are important for teachers to do which to me are progress of students towards college and careers standards, certain types of instructional shifts that we think are gonna be associated with success for students on that use of formative assessment data as a diagnostic tool. And in terms of the quality measures or observational measures that are used to compliment the student growth data, try to have those things be in harmony with the major priorities and then I think if there may be some purpose for individually tailored SLOs but I think there needs to be a common component for uniformity rather than something that is planned on just an individual teacher basis. I just don't see how that's viable. Actually, Laura, feel free to jump in here but we were trying to kind of talk about well, where are places where they are are figuring this stuff out and they have made the most of that to have time to think through all those steps that Andrew was just describing and maybe getting to the place of more uniformity is what you're seeing. Do you wanna talk about some of the brighter spots perhaps that you saw? I think as I talked about a little bit in my presentation I think that Colorado and Delaware are some places that are really moving toward that uniformity that teachers can still in some respects use the information that they're getting to direct their instruction but that it's also more of a objective sort of measure. So those are two places that I would point out but I know my look was a little bit narrow so I know that there's probably other places that are doing similar kinds of things. I know when I talk to somebody from Impact, Sam Piercy, he talked a little bit about D.C. kind of moving in some of that direction. I'm sure I can speak to that. Yeah, I wanna talk a little bit more about Impact because I'm curious about yeah, how you over time have iterations and changes. Right, so I guess in response to the pilot or not we didn't, we rolled it out and went with it. That is not to say that we haven't iterated on it every year. So I think that's actually more important than piloting is really being ready to take enormous amounts of feedback every year and to make changes. I think that's generally important but it's also how we've gotten probably the teacher and principal buy-in that we have gotten. They're seeing that their voice is heard as we make changes to the system. So that's just a point there. And we are actually now moving towards more standardization. So between years one and two we created recommendations, goal recommendations and assessment recommendations for every grade and subject area. Now some of those, for example, cosmetology would have been we recommend using a teacher created assessment. So that's not particularly standardized but for our elementary schools we recommended using the text and reading comprehension and we recommended particular targets for that. So we created guidance that we shared with teachers and principals and they could make decisions to use or to modify or not use that guidance. This coming year for the first time actually we will have mandatory literacy goals for teachers in particular grades and subjects. The district is really focusing in on literacy and we have aligned to our three reading, I'm sorry, our three literacy focus areas a particular assessment. And so where those are being used it will be 50% of their final task or SLO score. Which grade are you doing that? We're doing that in K through five. K through five grades. So there's one, I'm gonna ask one last question then we'll get to everybody in the room. And that is in a one unspoken assumption here throughout all of this is that student achievement data and its use in teacher evaluations is here to stay. We're going to be using data on how children are doing and how they're progressing and we're gonna be doing that even in those pre-K, kindergarten, first, second and third grades. So I just wanted to kind of put that out there to make sure that, see where we all sit on that. Do you think that there are some chances for the debate to turn in different ways? Or is there a chance that the percentage of which the evaluation must include achievement data is going to shrink or grow as this debate continues? So whoever wants to jump in first on that one. I've been up, but before I just wanted to say one thing on the piloting versus not piloting. Even if you take external pressure and external forces off the table, there's a terrible tension that states and districts feel between an urgent need to get teachers better feedback about their performance to identify the superstars and the chronic underperformers in a way that we haven't before and to do it well. And there's no easy answer to that tension at all. And I think it's easy for wherever someone lands for people to say, well, they lean too far towards the urgency side or they lean too far towards the get everything in place side. I think there's a thousand right answers to that. But the key thing is wherever you land on that, that there are mechanisms in place to say, well, not this is the system and we're sticking to it, but that we're continuing to figure out what works well, what doesn't work as well and how we can move forward. On the student achievement data, I mean, I think because we have figured a lot of it out as we've dived in, I think there are going to be adjustments that have to be made. I don't think anybody would disagree that we don't want systems with multiple measures but figuring out the right balance between those multiple measures and which actually provide more meaningful data and which provide less meaningful data both in terms of validity and reliability and but also in terms of how actionable the data is that we give back to teachers. So we don't have the right magic formula. I think that's where we're going to continue to see tweaks and trying different things. At the same time, I think there's a real worry that if we're five years out and we have 98% of teachers getting an effective rating, that when we know there's more variability than that in teacher performance, that we have a lot of unanswered questions. Like there's a big cultural shift that has to happen here. I think that's mostly on the observation side in really giving people both the skill to evaluate good instruction but also the will and the capacity to be able to sit across from a colleague and say, I really thought that lesson I saw today was pretty poor and I'm sure Kate could tell a thousand stories about how that plays out but also on this side, especially where we use SLOs, it's about an administrator being willing to say to that teacher, well, no, this isn't unacceptable. This isn't a rigorous target and we have to rethink it or being willing to say, well, the data say you didn't meet this target and what are the implications of that? There's a lot of human elements for as much as we try to make it a more objective system. So I think we have a long way to go but I do think it's here to stay for the foreseeable future. Yeah, I feel like it's a paradox. I did have the reaction and I think Kevin mentioned as, oh my God, when I read this paper I looked at some of the actual plans that are in place. People seriously carrying out the plans that they are but I do think, as Laura pointed out in your introduction, I think it's a step forward to move from just looking at teacher credentials to looking at the question of effectiveness and I think we should not go back from that and I do think that, as Andrew said, looking at how the kids are doing, it's pretty hard to rule that out and say that's not something that we need to grapple with. I think the technical challenges are unbelievable but I don't think we should go back and my prediction is that we won't go back to saying that's not what we're gonna be including in as a factor. I agree, I don't think it's going away and I don't think it should and I wanna just backtrack. I think what we did the year that I did it was the right thing to do. I just think that as we went through it I saw all the issues that we were going to be presented with in the future which is what struck me when I read the paper which is how I got in touch with Laura. I was like, oh my God, I lived part of this and it hit me in that manner because I could see where it was going because I do think we made the right move and it was a really difficult thing and I think that it is here to stay and I agree with your point, it takes a lot of bravery on the part of the administrator because you're gonna have teacher pushback, you're gonna have teacher union pushback and as an administrator you have to stand strong to those pressures. But the other thing I wanna point out that I think we're forgetting is that it's very, very important that we don't turn our schools into little pressure cookers. I know what happens just around state testing time and how much pressure there is for our kids and our teachers and even our parents and families around just that two weeks of state testing and if there's going to be so much emphasis put on this that teachers are gonna feel that's gonna trickle down to kids, that's gonna trickle home and so however this happens and it does have to happen, it's gotta happen in a way where we keep in mind that we're dealing with little people. We're dealing with very little people and very impressionable little people and we don't want our three, four and five year olds to hate school because they're going to school and it's a pressure cooker and so we really, really need to think about that. All right, so let me open it up then. We'll have a microphone. Oh, moving around and I already see a couple of hands up. So actually here, yeah. Our summer interns, honey, is going to be coming around with a microphone and like this one in the front. And if you could just say your name and your title. I'm Mark Sobolski, I'm a legislative education consultant. I used to work in Arkansas and I'm glad Mr. Krugly added finally at the end of this discussion the effects that this is gonna have on kids because I've seen it before teacher evaluations were even considered, SIG grants were considered and the intensity of the pressure is overwhelming. The principals have incredible pressure to create a pure result. Therefore, the little buddies, six and seven year olds have to stay in their room all morning long and not go to the bathroom because who knows, somebody else in another room could be in the bathroom and they could share information which would taint the entire process now, wouldn't it? Now, I'm going to a four hour discussion tomorrow morning at the Chamber of Commerce. I assure you will be allowed to go to the bathroom. This intensity, and no facilitator. I'm not able to go on a field trip. Yeah, there you go. No facilitator was even allowed to point out what page the test was on and they were thrown out of the building immediately. Can you get to a question, please? Do you think this is a good idea because this is already being played out and told upon the kids? Because that's what's going to happen, especially if teachers now are going to be evaluated even 10% based on this. And he's not leading the way, he wants you to know. What do you know? What do you think? No. Because one more practitioner in this group. So this is, yeah, it's the elephant in the room and I think it's an important, I mean, it's part of that data question that we just had at the very end, how we're going to use it if it's here to stay. Really, if you like to just tackle, it goes to that pressure cooker point. Call us a little further. Again, it's balancing competing needs where there's no one right answer, right? I don't want six and seven year olds to feel so pressured or that their teacher's job depends on their performance on a test or that their school building closing depends on their performance on a test, right? I don't think anybody feels comfortable with that scenario. At the other end is I don't feel comfortable with a six or seven year old getting a bad education that is impossible to recover from if they have a poor teacher several years in a row or in a school that's been low functioning for years. So we have to, what you're saying is a real concern. I don't think we can disregard it, but again, it's figuring out how to find the best balances of these competing things so that kids ultimately benefit. You were a practitioner, don't you think that would have an effect on you on a day-by-day basis, not just on testing, but on a day-by-day basis? I mean, yes and no, I think, I think when we talk to teachers in Tennessee, right? Tennessee was one of the first states to implement statewide and the first year the newspapers were filled with stories. Every place you went, there were stories of teachers the teachers throwing up on test day, let alone the kids and in the emergency room and then the second year, which is this school year that just ended, they'll be going into their third year, there were way less of those stories. I mean, so much of it is more fear of the unknown and then fear of what it really is once you see it. So I think we have to get past some of these hurdles and work our way through them. I taught for a long time, we did a ridiculous amount of test prep and that put pressure on the kids around the test. And you did as a teacher, you did what you could to mitigate that. We have to be able to move forward when that schools are putting, it's a leadership problem when that six hours of a day is required to be spent on test prep. That we can do things about these things and move forward. I think it's also important to note that even if we were using the old method where the principal is just going in and just observing practice, there would be teachers who would throw up or get sick that day too, because that's a scary pressure filled thing for the teacher as well. So I think that that's important to note because just that, you know, what you said is we wanna make sure that all of our kids are getting the best education that they can. And I do think that we need to blend these two things together. And Lisa, you wrote a beautiful paper on teacher evaluation and the observation piece of it. And I think that the blending of those two things is very, very important because we need to look at teacher practice as well as student outcomes. I think we need to look at both of those things together. But in looking at just teacher practice, teachers get sick and freak out too. Yeah, I would agree with both Sandy and Andrew. And I think that it's why it's so important to for the ongoing evaluation. And I think, you know, piloting when possible before adding some of those stakes to make sure that kinks are worked out, to make sure that it's as fair as possible system. But I think, you know, you're, as Andrew was saying, you know, teachers are gonna feel some pressure regardless and so finding that balance and trying to, you know, mitigate so the kids or the children are, you know, not affected by that as much as possible. I think comes down to that, that stakes issue. Let's go to another question. I see a hand up in the back, I'm in with that green checked shirt. Thank you. Hi, my name is Teddy Hartman. I am a Education Pioneer graduate fellow at the moment. I was a teacher for a decade and now I'm doing ed policy work at the University of Maryland. So my question is looking at the kind of, the risks that's resource intensive and that teachers and principals might not have the training. I'm wondering if there's any kind of work to be done with by state by state with teacher credentialing programs or as kind of teachers are coming out getting their certificate or graduate degree, whether we can shift some of that so that part of that would be learning how to kind of norm these SLOs and kind of having some of that training so it's not just school site specific but kind of state by state so that there's some of that kind of burden as you're saying shifted on somewhere else in the system so that kind of coming out with the credential. Teachers could be a little better qualified for some of these things. So yeah, I know that for sure Laura will want to comment on that. That's something she's been doing some work on writing about but any of you want to jump in first on the question? I would just say that we're trying to work at the Chief State School Officers with a group of states on kind of doing a variety of things to improve the pipeline for improving educator effectiveness and looking particularly at levers that states have a particular role in including how to set up better certification standards for entering teachers, program approval for teacher education programs and better data to be able to track what happens to graduates of different teacher prep programs but I think the example that you touched on in particular of doing a better job in teacher preparation programs around the whole issue of assessment literacy, how to understand the purposes and limits of different assessment tools and how to use assessment data in a diagnostic way all would be terrific kind of priorities for improvement. So I thought you made a great point. We do a lot of work looking at teacher preparation programs. We released our teacher prep review in June and there's a real disconnect between what programs do and what the state has put in place for what's gonna be the requirements in the classrooms. And it's a disconnect I really personally, in fact, Laura and I were talking about this right before we started, right? There's this disconnect between the idea of academic freedom on the one hand and but you're a teacher preparation program on the other hand, right? You've asked to be a teacher preparer in the state and there are a set of requirements that go with that. I think you're exactly right. If a state has a model evaluation system in place, wouldn't it behoove every program in the state to make sure that its graduates are familiar with that program? But both on the observation side, on this SLO side, it just enhances the pipeline. It makes it so much easier for the candidates to make the transition into the classroom. But it's a really heavy push to get programs to recognize that that is part of their responsibility. I think through CCSSOs and other efforts, states are really getting ready to be a little more demanding of programs and it's to the benefit of the teachers and the kids, I really think. You know, the comments that I was gonna make have already been said, so I would echo what's been said and reiterate the communication point between what's really needed in the schools and what the state's required and what's actually happening in teacher preparation programs in my previous paper that I did called Getting in Sink, looked at early childhood preparation programs and elementary preparation programs and just that they're not really meeting the needs of what's in the students' classrooms or this issue of assessment literacy or how to use data that they get every day on their children and use that in a way to inform what they're gonna do the next day. It's just, it's really not happening. And to that point, the comments the gentleman made earlier that were very salient around kind of what does this mean for those little kids, I'm almost wondering as well, within our teacher preparation programs, there needs to be a sense of developmentally what it means to tell a five or six-year-old this test really matters. If you want to, how to help teachers understand how to communicate to their young children in their classrooms without suddenly putting a extra level of pressure on them that young kids at that age may not even fully understand. I think that issue of the developmental sciences is really missing from most teacher preparation programs, especially elementary and when you think, and in most states there's teachers who could get an elementary license, which from elementary preparation or an early childhood license and still teach in K one and two. So I think early childhood programs do a little bit better job with the developmental sciences and being able to understand how children learn best in those grades and how to talk to them about different issues so they don't feel all the extra pressure coming down from them. But it's really absent in the elementary and teacher preparation programs. So we have teachers that are working with these young children, but aren't necessarily understanding of the best ways that they learn at those young ages. And so I think that's a huge concern. Area for improvement. Okay, here on the right, a gentleman in the salmon-colored shirt. She did my colors right today. You're doing a good job. This fellow as well. Can you speak up just a little bit? My name is William Jackson. I'm an education pioneer's fellow as well. And so my question, I guess it's two parts, but I try to make it really brief. In terms of switching, I think Mr. Crickley, you said something earlier about target versus growth goals. And so like how the conversation is changed to be more about growth of this particular teacher. And so does that help kind of ease some of these pressures that teachers feel that they don't have to be at a certain target, but they're judged on how much they grow. How much the students in their classroom are growing. Or do you mean also teacher growth? When I said it, I meant the students growth. Yeah, and so I mean moving that idea to, when we assess teachers, we're talking about teachers growth. So this is where you were at the beginning of the year and this is where you were at the end of the year. And you're taking these steps to get here. And so this is how we're gonna assess you. As opposed to, oh, I'm at this target point. And so now I'm good or I'm not at this point. And so I'm a bad teacher. And so how does this kind of become a more formative conversation? And so people are pushed to want to be better and not just to be at a certain point. And then I think the second part is, is how do we hold these administrators accountable for the teachers growth? Because you're continuously putting this pressure on the teachers and the students to perform. But then at the end of the day, the administrators are just sitting there like, oh well, you know, I'm doing the best that I can. And nobody, so how do you hold these people accountable for trying to develop these teachers? And how does this kind of all fit in with the SLO thing? So there's a lot in this, in those questions. Thank you for that. I would love to hear also, Kate, on impact the principal piece of it perhaps. But do you wanna go ahead, Andrew, and respond to some of that? Well, I'll respond to the second part of it. I had a target as the teachers each had a target. I had a target too. So I had a target in terms of, I had a target in terms of the number of students in the building that had to be above grade level that at the beginning of the year and at the end of the year. So it was not just the teachers that had a growth target. And what was the repercussion? What was on the line for you? The repercussion was, it had to do with your evaluation rating. And did that lead to salary adjustments? I was kind of curious what the stakes are. I know that a lot of principals feel this. Well, in Illinois, principals are not tenured. So you could, in fact, lose your job. The way our contract was written is if you get a certain number of unsatisfactories in a row, you could be fired. So yeah, a personnel decision could be made on that. The first comment, the first question, I don't know that I'm the one to answer, but the second one, yeah, I had a goal too. Well, in terms of the school administrator evaluation and how they're held accountable, school administrators in DC are evaluated under impact as well. They have a, their impact looks slightly different from the staff impact. There are two main components. One is around DC CAS, our standardized assessment and hitting goals that they've set with the chancellor at the beginning of the year for the DC CAS. And then the other part is around what we call the leadership framework. And so that's six core standards, instruction, school culture, but then particular, talent management. So whether or not they're retaining their highest performing staff, and fortunately through impact, we're able to identify our highest performing staff members. So we know who it is that those principals are targeting for particular retention. And so a few other things go into talent management, but principals are held accountable through goals and the leadership framework. And ultimately we have a performance-based pay system for our school leaders, and they are subject to separation. They're also at will. That was the term I was looking for. Yeah, so they're also at will, but their evaluation is a part of the chancellor's decision as to whether or not to renew contracts. In terms of the evaluation system being a growth measure for teachers, we certainly hear that from teachers. We've heard from teachers that since impact, this is the first time that they've actually had observations and real conversations about their teaching from their administrator. And then we also, as part of impact, have what we call master educators. So they're an outside objective, third-party content area expert, and they receive two observations from those folks over the course of the year. And we hear regularly that it's the most robust feedback they've ever received in their entire career. So that actually really is helping them grow. We also see, we're able to see the teachers who are moving across the ratings category. And we're seeing that a large number of our teachers who were rated in one of our lower categories minimally effective, move to effective. That it actually is, they're getting feedback that is helping them to improve their practice and move into the effective category. And that actually, it can be something of a catalyst to help prompt them to reach out and to get some of those supports. So it certainly is an improvement mechanism for teachers. Let's take just a couple more questions, and then we will wrap it up. There's, I know what it will do. Let's do kind of a rapid fire round. I'll take two questions together and then we'll answer those together and I'll take two more together and we'll answer those together and then we'll have some closing remarks. Two questions at once. I know. Mixing it up. I'm Cara Jackson. Like Katie, I'm also a doctoral candidate at Maryland and I'm about to become the assistant director of accountability for the Urban Teacher Center. And so one of my questions for you is all of this teacher evaluation reform is sort of coming down the same pike at the same time at the common core standard. So are you aware of people working to sort of create these student learning objectives with the common core standards in mind, with the idea in mind that you could maybe have economies of scale where states are sharing their SLOs so they can kind of combine information. Good, okay, so we have a common core alignment question and yes, go ahead, if you can grab the microphone and introduce yourself. That was my exact question. Hey, hey, help me do that. All right, yeah, let's grapple with this one in terms of that kind of connection and that knowing of course that with the common core standards also come in the common core assessments that would be testing those standards and how they'd be used in the early grades which are reaching down younger than third. Who wants to jump in on that? Or I know, Laura, this is something if you want to give us some of the factual. Well, I know that the states that are implementing common core, the standards are guiding the SLOs, but as far as to the second point about states working together around developing common, I don't, I'm not. I haven't heard that. Yeah, yeah, I wish I did too, I don't. I mean, I would just throw in quickly that there's a really good paper by Aspen Institute on the importance of kind of tying these initiatives together that would be worth looking at. I gotta copy it with me if you want the, I can't remember his name, but also I think there's some real technical issues around the timing of introducing the new assessments that are key to the common core and then when that data comes into play in terms of cycles of teacher evaluation and are you gonna be using kind of tests based on the old standards to render consequences to teachers and sorting that out, I think it's led in some ways to kind of the desire to kind of put off, delay the onset of these initiatives or put off high stakes consequences and the extent to which that could be done so that you're really lining up the impetus to promote the common core with the incentives for teachers and not have things get crosswise. Maybe Laura, this is something you've encountered as well, but has there been any case where a district or even a school deciding on SOLs has said, well, we're gonna do this as an interim measure until we get the common core assessment and then we'll use those as our, or we'll somehow create SLOs out of those or has that come up? I mean, the assessments that are coming online aren't necessarily like, as far as the standardized part of the state systems aren't reaching down into the early grades. I mean, I think it was Park that was looking at creating some formative kinds of assessments for teachers to use in the early grades, but I believe that they've put that on hold so that's even more delayed now. But I think, I mean, the only, there are some interim kinds of things, like we'll use this until Florida, for example, a lot of the districts are using that shared attribution approach until assessments are created for every grade and subject area. So that's sort of a place that it's being interim. I think there's also a lot of places, like we heard earlier, that are using an SLO type model for all teachers, even those in the tested subjects, but it just counts more in the early grades. So, okay, DC. So, two more, we'll take them together. Here in the front, these two folks, Albert, or yeah, go in the front and then Albert, thanks. I'm Anne O'Brien, I'm with the Learning First Alliance and my area of expertise is not in early childhood, so I'm kind of trying to wrap my mind around some of the things that are being discussed here. I'm curious as to whether there has been a lot of pushback in places where these types of especially standardized systems of evaluating, especially the pre-K and kindergarten students, pushback from parents, especially middle and upper class parents and their views on kind of this whole trend of evaluating these younger age teachers using student achievement data. And then I'm also wondering as, particularly, maybe regards a literacy test for early grade students, how developmental issues are, child development issues are taken into consideration because it's my understanding from my limited knowledge of the research that there's so much developmental difference among kindergarteners, among first graders and among second graders that it really isn't until you hit the third grade that you can kind of start separating out some of those characteristics. So, great. If you could just talk about that. Excellent. That's actually very related to what I wasn't gonna ask. It's sort of another elephant in the room that I think the only thing we've talked about, so this is a really big room for two elephants at least. So Laura in her paper talked about the unique attributes of pre-K to third students and the way in which they learn and demonstrate the learning. And so the underlying assumption of all this is that you can actually reliably measure growth in kindergarten, first grade preschoolers in a way that you can tie high stakes to it. You know, student learning outcomes sounds great in terms of like, really could provide rich PD, professional development opportunities. But we haven't really talked about sort of the underlying assumption that this is we can get reliable and valid data that we can then tie high stakes to teacher evaluation. So we'd love to put the photo panel to address that. Okay, thank you. So I'm just gonna, so to reiterate those three questions and given our time, I'm also gonna ask our panelists to, if you wanna make a closing statement as well within your answer, please do. So we have a question about this. I've been parent pushback in anything that you've seen around this. Do we know developmentally as children are regressing from say five to six years old to seven and eight, is it, can we in fact measure their growth and it kind of pertains to what Albert from National Governors Association, thank you Albert for being here. Albert Watt had asked about, in terms of just validity and reliability of the assessments that are being used in the younger grades in the first place. So let's do this. I'm gonna actually put you on the spot time to start with you and go all the way down and then have Laura close us out. So I'll try, I'll address Albert's question, which I think is a killer question. And it's, this is such a tough issue because I think my perspective, as I said earlier, is I think this is the direction that we're moving in and I don't think we're gonna go back. I think the technical issues, even with older kids, most of the best methodologists are pretty dubious about the value added method in terms of can you reliably identify these kind of outstanding or lower performers, or do the scores bounce around for factors that are not related so much to the teacher effectiveness? And it seems like the verdict from people who are very serious about this stuff and know a lot more about tests than I do, is this doesn't work. It just feels like we're moving in this direction in spite of that huge elephant. And I think for younger kids where we have less of, we have less experience technically with large-scale assessment and measuring growth, they're probably more serious problems. So it's a huge dilemma. Thanks for raising that. I'll hit the parent one. I think that it's really important to explain that, bless you, bless you. I think it's really important to note that if a school is using the data that they have appropriately, they're using it to make instructional decisions. And so the amount of data that comes in shouldn't just be for SLOs or for teacher evaluation, it should be for instructional decision making. And when that's explained to parents in a manner that is real to them, that they can see, here's a data chart, here's a dashboard with unidentified, here's how I start grouping children, here's how I start planning for instruction, then the amount of assessment, it doesn't become an assessment anymore, it becomes part of instruction. And so then there's less pushback because they understand that it's every day and it's used every day. And I think that just in closing, I'd like to say that I fully agree with one of Laura's points near the end of the paper is that whatever happens, that it needs to be aligned. It needs to be aligned from the earliest grades through the later grades, all grades, pre-kindergarten, even if we're gonna start assessing the early childhood teachers zero to three, needs to be aligned and it needs to be for all of the subjects, not just reading math. And I thank you for that wonderful paper. So just one, the only place I'm aware of with parent pushback and it wasn't around student achievement in Georgia, where they had planned under their race to the top grant to include student surveys as a counted part of teacher evaluation at the K-2 level, parents really just couldn't wrap their heads around that how their five-year-olds were gonna circle a smiley face or a frowny face and that that was gonna be part of a teacher's rating. Teachers weren't too crazy about it either, but the parents really didn't like it and they ultimately dropped it and have had some interesting fallout with the feds over that being because it was part of their plan. I'm not sure to what extent, I mean like we all kind of think like, yeah, that's kind of a hard sell, but I'm not sure it was well-communicated. Student surveys in general are not what first pops into most people's head, which is, do you like this teacher when they first hear about it? So there may have been more communication. I think to the other question about how are we making sure this is valid for the early grades, this is where really our emphasis on growth is really our friend, right? We're not saying did every kid get to this proficiency level, which a kid might get to in kindergarten or might get to in first grade and there's really not a developmental problem at all. It's just, or a learning problem, it's just kids are different at those ages. But what do we want to make sure is happening in this classroom and how are we going to be able to get out of I think our important questions to ask and we have to keep that in balance with making sure that our measures are fair and accurate and I think these grades are gonna continue to be a challenge. At the same time, we can't risk saying, nah, kindergarten, first grade, what will happen will happen and we'll sort it out later on, which I actually think is the plan in many schools. At the school that I taught in in Brooklyn, we had a principal for a while who tried to lure the best early childhood teachers to the tested grades and placed the teacher she saw as really weak into kindergarten and first grade where they can't cause any problem. Really? So it's how it all fits together into the whole equation. I would say in terms of the reliable data or valid data or how child development plays into this, my gut reaction to that is that this is a complicated issue and there's a lot of really critical professional judgment on the part of a school leader and on the part of a teacher and to trust that they really are instructional experts and that they have the content knowledge and the background to make smart decisions around goal-setting in the beginning and then reviewing the data at the end. So if we had a situation where perhaps a child had a traumatic event happen over the course of the year, which would impact then potentially their score at the end of the year. We've had teachers who call and explain the situation and we ask them and we talk with their administrators to have a conversation about that as they are looking at the data at the end of the year and take that into consideration. Use your professional judgment and that's okay. That actually works and I think is a really strong outcome that people are having real meaningful conversations around this data and around the assessments. So I just add that I think that the point or the little anecdote that Sandy mentioned is really important here and I was a former teacher and experienced had some of the same kinds of things happening in my school and I've heard that from other teachers that the teachers that aren't getting the scores and the older grades that the principal wants are moved down into kindergarten and first grade and so we need strong teachers everywhere and we especially need highly effective teachers in the early grades to really set kids on the right path and getting that right start. But the issue that Albert raises is really a huge one and I think it's also why the focus on multiple measures is important and so this question of what weight do we put on these different components? Observation is an important piece also and finding ways to, I know in Lisa's paper watching teachers work talks about different tools that really look closely at teachers interactions with students which we no matter so much and the kind of language and conversations they're having with students and so I think the more that we can look at those kinds of things and then also how it's important to really examine or observe how teachers are using the data that they're getting from any kind of assessment whether it's the ones that they're doing every day and the formative ones are using every day by just asking their students questions and seeing what they've learned or didn't learn from previous lessons. I mean how teachers are using that information is an important piece as well so I think it's a question of balance and so that's what we really need to think about and look at and that's to answer the question posed earlier I think that's really where the debate's gonna go next. So thank you to everybody up here. Please join me in giving everyone a warm round of applause for time.