 and is now being recorded. Hi and welcome to today's webinar entitled ESTA Implementation, The Role of Evidence-Based Practices. We're delighted to have Kyle Snow and Chris Dwyer from RMC Research Corporation join us today. Before we get started, there are a few things I wanted to share with you. First of all, that the audio portion of this event is only available by telephone. When you logged on, the platform probably asked you whether you wanted it to call you or whether you wanted to call it if you chose to call in. The number again is 1-877-423-6338, and then you'll be prompted for a passcode. That number is 142587, followed by the pound side. If you have any difficulties beyond that signing in, there's a 1-800 number. You can call 1-800-945-9120. I also wanted to briefly orient you to the platform you're using today. You'll notice that the presenters are available by video in front of you. Your video is activated, but you don't have to make yourself visible if you don't want to. Below that is a chat section. Please feel free to write any comments or questions that you have in there at any time during today's presentation. We really encourage informality and discussion, so feel free to jump in that way at any time. Then on the left, you'll see that we have a PowerPoint presentation to accompany today's conversation. Again, let us know if you have any questions or comments in the chat box. Thank you very much. Let me tell you a little bit about who we are. The Appalachia Regional Comprehensive Center is a member of the Comprehensive Center Network, which you can now see from the new Comprehensive Center Network portal available at the URL listed here. You can learn about the breadth of work that we do across the nation. And as you can see in the map in the lower right corner, the Appalachia Regional Comprehensive Center serves Kentucky, Tennessee, Virginia, and West Virginia. We're funded by the U.S. Department of Education to provide technical assistance to State Departments of Education. I'm delighted to introduce today's discussants to you. Kyle Snow and Chris Dwyer are both with the RMC Research Corporation. Kyle is a team member with the Appalachia Regional Comprehensive Center, and both Kyle and Chris also work with the Mid-Atlantic Comprehensive Center and Chris with the Northeast Comprehensive Center as well. But I really like to give them an opportunity to share a little bit about their background and their interest. So I'm going to turn it over to Kyle and Chris to tell us a little bit about their background before we move into the content of today's conversation. Kyle? Sure. Thank you. Okay, let me welcome everyone to today's webinar. My name is Kyle Snow and I am with RMC Research Corporation, as Ken said. My background is really as a developmental psychologist in child development. But with that background, I took direction of heading into education policy, particularly relating research to policy and the topic that we're with today. And Chris? Yes. I really think of myself as a school person. I spent a lot of time both working with schools and as a teacher myself, particularly in early childhood and elementary school, and then really moved into doing program evaluation work. Probably my most relevant work for today's topic is the seven years I spent working with the IES practice guides. I was part of the team that took the practice guides and turned them into videos and tools for the Doing What Work site. So this notion of practices that have strong evidence, moderate evidence, weak evidence is very close to me, especially in literacy and mathematics. Thank you both very much. Sorry, it's open. Caitlin, or do you have any more words? Yes, no, go for it. It's all yours. Okay, go ahead. So thank you. And see the final view is an overview slide. I do this as much for me as anybody else. It's beyond track. We have a few points we want to try to touch on a little bit here. And we're going to try to vary our pace and respond to questions and things that come into chat box. So please let us know if we are going too quickly or too slowly, and we will try to adjust on the slide. So it's tricky when you can't see your audience to figure out how you're doing. So I'm going to take advantage of these typing. We're going to start today with just a real quick second talking about the important role that evidence has played in U.S. education policy in the recent past. We talked about under the current law the use of defining a series of evidence and then go from there into how do we identify what really are evidence-based practices and activities under these series of evidence. Chris is going to talk a lot about what's called Tier 4. Tier 4 as we'll get into is a really unique part of the way that evidence is being talked about in the current education law and is a great opportunity for states and local school systems to really be creative and innovative. And then of course we have the inclusion questions towards the end there. So that's kind of the overview of where we're going to go. And again, this is not an exhaustive coverage of this topic, a tiny stretch, more of an introduction to it to get us all into the center so we can dive into some really more additionally challenging issues later on. So the role of evidence in U.S. education policy is not new. Of course the rule governing education policy, at least just in the K-12 space, is the Elementary and Secondary Education Act. And the current instantiation is the Every Student Succeeds Act. Some of you may know this in its prior version called the No Child Left Behind Act. Of course you can notice these are all kind of the same things, the same overarching law that governs how the federal education department interacts with these states and then in some ways how the states interact with the local agencies. And in both No Child Left Behind and the Every Student Succeeds Act, the current version, there is a real interest in bringing evidence into, or research-based, into the way that we do education in this country. The idea, of course, is that it's better to think about spending billions of dollars to have some idea that it actually is likely to be effective or work. Whether it be through research or evidence-based or whatever words we're going to be to define it. There's really an interest behind No Child Left Behind. And the No Child Left Behind version of the bill talks a lot about scientifically-based research. That phrase, scientifically-based research, many people did some counseling about it. Anywhere over 100, 110 times that that show up in the law. It's a very big part of No Child Left Behind. But really what's scientifically-based research under No Child Left Behind meant is the use of randomized controlled trials. This is a sort of research design that is really very simply kind of where we test drugs. So you take a drug off the shelf and you assign half the people that want to be in the study. You can keep the drug and have to now keep the drug. And you see how effective it is at the end of the day. That's like the shortest definition of randomized controlled trials ever. But you get the gist of it. And that was really the crux of how No Child Left Behind saw research. For folks who are in research or who write about or read about or think about research, random clinical trials are kind of the holy grail of research designs. It's the top most demanding design that there is. And so as a consequence, the research base in No Child Left Behind was really focused in on this sort of higher, very top level of a piece of the research puzzle. And we'll talk about that more in a little bit. Under S Under Every Student Succeeds Act, there's been a shift to this phrase evidence-based. So it's kind of taking the same role as scientific research to give us some scientific undergirding to our practices. But instead of being scientifically-based research, this phrase evidence-based shows up. And importantly, instead of just focusing on random clinical trials, ESSA really focuses on four tiers of evidence-based research. Again, we'll get into what those are a little bit later on. The key point is while random clinical trials are still part of the mix here, ESSA really recognizes a broader definition of research to support programs. So while there is the most rigorous type of research design included as part of the evidence base, there are somewhat less rigorous research designs through the various tiers that we'll talk about. One of the things that this does is provide much more flexibility to states and local agencies to really match what their needs are with what their evidence to suggest would be effective for them without needing to rely on their being that clinical trials kind of study. One of the real limitations of limiting your definition to clinical trials research is that that's only a small proportion of all the research that gets done. So it's really an awful lot on the sidelines by focusing only on that kind of a design. And the four tiers of evidence-based research really opens up the definition to include this broader breadth of evidence. One of the things that happens when you do that is instead of someone saying to you, this is what constitutes the best research evidence that there is available, it really requires everyone to develop a sort of culture of evidence or critical evaluation of research so that across the various tiers of evidence-based, not all studies are going to be created equal by design, but also the studies that are going to apply to the problem or sort of challenges the needs identified by schools locally in very different ways. And so the idea is that by giving the flexibility, recognizing the broader range of research, there is this emergent idea of culture of evidence that should be driving our decision-making as school systems. And I mentioned the role of research in ethnic or graphic or visual person. If anyone who ever looks in my office will attest, there are four tiers of evidence that are shown here in the pyramid in the middle of the screen. You will get into what all these are in a few moments. Important elements here is research recognized in a known child of suffering as indicated by the box on the left. Really, in this top box, there is strong evidence of what ESSA refers to as strong evidence clinical trials research, whereas the research that can support practices under ESSA really exists across all the tiers of evidence. Now, within ESSA, research or evidence-based shows up in a couple of different ways and treat it in different ways really. There are some aspects of the ESSA law that require that practice is to be evidence-based. And so we're going to start with those. The others will follow on the next couple of sides. For the most part, what this idea of requirement means is that states have to provide funding to certain programs, and that program funding must be evidence-based. So states are given money from the U.S. program of ed. That money is intended to be spent on programs of a ticker type, and those programs must be supported by the evidence, but must be evidence-based. Particularly in Title I, for those who get into policy talk, there are some notations that will see sacrifice of various sections and things as for your benefit. If you're not into that kind of thing, then you can just sort of gloss over that a little bit. Don't get too tied up in it. Title I, Section 103, includes 7% set-aside. Again, that's a formulaic piece for policy speakers. That's a pretty big chunk of change under the funding that's targeted toward school improvement. You may have heard about school improvement projects, improvement grants that comes out of this title, the 7% set-aside. So under Title I, states must fund programs to help improve schools that are struggling, and these programs must be evidence-based. It's also a bit of this requirement and requirement that programs be evidence-based in Title I under parents and children engagement provisions. Other positive aspects talk about allowing evidence-based, and in these provision states and local agencies are encouraged to invest funds in evidence-based activities if they are funding those activities. So unlike Title I, then requires certain things to be funded. There are other aspects of S that were states and localities had the option for funding programs. So for these provisions, if states and localities choose to fund certain activities, then there's an encouragement that they be evidence-based, and the state to build that in has to demonstrate there is some recently available evidence to it. And so that shows up in parts of Title II, which is where the teacher training, preparation, and recruitment pieces, Title IVs, those student support grants, and also some of the competitive grant programs. Finally, there are elements of ESO where these are evidence-based and incentivized, meaning states aren't required to fund certain programs, and if they do, they're not really required to use an evidence-based, but there's some kind of incentive to do that. So kind of the optionals give you extra points in the scoring. This is typically true of a lot of competitive grant programs. Some of those are listed on the screen here, the teacher grants. They were extremely engagement-centered, learned, and so on. In all these cases where the competitive grants are incentivized, proposing to conduct programs or interventions that fall within the top three tiers of evidence give you a competitive advantage. I haven't told you what those are yet, but that means having some degree of strength and evidence will give you a competitive advantage to getting a program funded. Thanks, Kyle. So we're about to dive into understanding how ESO frames these tiers of evidence. But before we do that, does anyone have any questions or observations at this point? That sounds like a no. Okay. Thank you. Okay. So these tiers of evidence are four tiers of evidence, and very obvious questions are, how are these defined? These are defined in the law itself, so it's not, you know, sort of trying to figure it out. The law actually provides for a pretty good definition of what these tiers are. As I said, it includes the sort of consideration of whether the existing research or existing research studies of various kinds that tiers once a tree, but also includes this tier four, which we'll talk about more and more in depth in the latter part of the talk. Which is not so much that there are a whole bunch of research studies about an intervention, but that there is a capacity to build that research base within the law. That's a real key difference in how evidence and research is thought about under ESO than no child left behind, kind of research or evidence building function. The first three tiers, ILC, again, we'll get into definitions in a moment, really provide more flexibility and options to meet local needs. So each one of those tiers, one through three, has some evidence behind it. The evidence isn't varying strength according to different definitions of research design and so on, but by broadening the pool of potential evidence, you actually have a greater likelihood of having some evidence in forming what you're doing rather than relying on, since there's no tier one research base, then you're kind of stuck. So you have the option to apply a broader range of research to meet local needs. What that does though is opens up the door for potentially contradictory and consistent findings when there are a lot of different studies at play that gets really frankly very complex and very confusing and really requires more of us in education to be more critical consumers of what we're trying to do. So states must determine under ESO, must determine how to manage the identification of intervention at each of these levels of evidence. That's going to be one thing we'll talk about in a few moments. So under this tier four, really as I said, it opens the door for innovation. So these are a space for a locality or a state to say, hey, we think for various reasons that we'll get into that this isn't going to work, so we're going to implement this. But then we're also going to take advantage of that implementation to do some research while we're trying this out. That's really an opportunity for states to build and contribute to the research base. And importantly, states can choose to pursue this function or not. And so that's required. Chris, I'm going to talk a lot about why this is a wonderful opportunity for states localities to contribute to the research base of what works. So how do we define these four key observance? Well, I'll give several characteristics. I'm going to hit all of them. We're going to hit a few of them just really very briefly. First is research design. So what is the actual construction of the research study that is coming into play? One of the things that's important to note is that research design is really in the law defined within the space of quantitative research studies. Those studies are really numerically based as opposed to qualitative research studies. It's not to say that qualitative research designs cannot be very, for a rigorous and very rigorous system of law, happens to be written. The research designs used here reflect a lot of the convention and ideas that are found in the What Works Claring House, which came out under No Child Left Behind. Sample size of the study also becomes part of the characteristics identified that it's a study into insurance here. So speaking, research that has been built in large samples are given a more, considered given a more rigorous, more rigorous. Probably this is because they oftentimes include relevant subgroups. They also provide different types of statistical advantages. So when we talk about the various tiers, you'll see some reference to sample size being an issue here. One of the things that focusing sample size really challenges us is that some of the things that we might see in education just or not don't lend themselves so well to very large-scale research because the phenomenon, although it's reasonably common across the country, may not be very concentrated and allow for a large sample size study to take place. So again, there's a little bit of a challenge in meeting that definition. And finally, the strength of the effect. So a research study is undertaken to demonstrate a relationship between a couple of variables and a basic idea. And the effect can be strong, weak, or non-existent. In the parlance, we talk about our statistically significant effects. So these are effects that really we have confidence are real and due to what we think the study is telling us they're due to. And this effect ought to be, under these terms of evidence, an irrelevant outcome. So if you're looking at interventions to improve literacy, a greater sense of self-efficacy among students might be able to hold an outcome but it's not a relevant outcome to the intervention. And so it has to be looked at in a slightly different way. An important element of how all this works is, I would apply it here in quotes, in my shorthand, the law's shorthand is the at least one study rule. And so the idea here is that if there's at least one study that meets the highest tier criteria, then that study represents the highest level of evidence and says the most evidence-based, if you will, study that's out there. When you open the door to consider all sorts of studies, multiple studies of different types of designs, what you find is they're very seldom. It's just one study. So this at least one study has met its highest tier of evidence-based. It becomes really critical to think about, especially as we're looking at the possibility of having multiple studies, again potentially with conflicting findings. So graphically we can show what the difference between these four tiers are in this table. So we have them layered up by the names in the law, strong evidence, moderate evidence, promising evidence, and then demonstrated rationale. And it's going to highlight a couple of the differences. So on this screen you see that across these first three tiers of evidence, the nature of the research design really differs with the strong evidence level of evidence really being driven by experimental study. So this is randomized control trials very much like was described under Michelle F. Mahourin. For moderate evidence, interventions of programs quasi-experimental study, studies can give you more kind of evidence. So you may not have random assignment of students to condition. You may have all statistical controls that you might have in an experimental study. It's still a pretty vigorous study. We have multiple groupings and you might do some other way of controlling for groups. The promising evidence tier is really where correlational studies fall into place. So correlational studies, you're not necessarily controlling one variable to see its effect on another in any way, shape, or form, just like it was naturally occurring. You can put statistical controls on that study afterwards to help strengthen that study. Again, this kind of represents, if you're putting in a classic criminal design, the sequence from very vigorous research design to much less vigorous research design, but still a research design. And that's important points to take away that these levels of evidence are all research-based in one way or the other. Again, there's two other points about sample size and statistical effect. Those show up across all of the top three tiers of evidence, except for tier three, the promising evidence. Not that large samples are not a good thing. For correlational studies, it's just that within that tier of evidence, the expectation of having a large sample is not set to the same level as in the top two tiers. And finally, you have this sort of anomalous group down on the bottom here, demonstrated rationale, which is different in name, but also in description. This level of evidence describes a set of studies or a situation where there's some kind of really well-specified logic model. There's some reason to think that this is going to work from research and other experience, but there's not a particular set of research studies out there to draw from, but there could be. And so this is considered a tier of evidence for many aspects of ASSA, not to say for all of them, but for some of them. Again, we're going to come back to that at the end with a lot more deep dive, so I'm not going to go too much into that at the moment. So you have four tiers of evidence. They vary a little bit in terms of their experimental design. But all of them, certainly once or three, everyone would agree constitutes different types of research. And even tier four has to have some research base rationale behind it, or there's some research out there that contributes to an idea that it should work. So with all that in mind, all those various studies floating around, and in some cases there may be very few studies, very many studies, states have to figure out how to manage the identification of the interventions that might fall at each level. So you might be looking at wanting to really turn around this very low-performing schools and urban schools or rural schools, let's say. Let me find all the researchers out there who might find a handful of studies on efforts to turn around rural schools that are struggling. How do you know what, which study carries the most weight, which could do the best evidence? And in general, what we're seeing now with Chris and I working with states through the Compt Centers is a couple of different models show up. In one case, as you see on the left, the state agency may provide a list of evidence-based options, and sort of say, here's where we figured out where that option falls, and go ahead and choose what you're going to do. Very simple, very straightforward in many ways. Very similar to the way under which I'll look behind, where once something was expected or deemed to be a research base or scientifically researched base, take a little sticker on it and you can very readily identify it. So it's a very nice way of sort of sorting out the universe, I guess. But we're also seeing, in some cases, states are providing guidance to the local agencies to identify evidence-based options on their own. So rather than give a very circumscribed, sometimes, list of options, the state is providing some support for the local agencies to really look at the research themselves and construct their own sense of what the spheres are ever to look like and make a choice between those two. So now, probably not surprisingly, we do see the possibility of there being some overlap where states are doing some combination of providing some list for something possibly and providing some support for how to choose amongst options for our locals and other cases. So that's a really quick glossing over, but it's really a very complicated process. Let me complicate it a bit more, and then we'll get some questions and some reactions. So you have to identify these activities, and regardless of whether the state is constructing a list or the local agencies constructing a list or there's some collaboration between the two, I think there are these four really critical considerations that come into the conversation. The first is the breadth of evidence and the breadth of research that comes into play. As I mentioned earlier, by opening up the definition of evidence-based to a wider range of research design, you hopefully have the door to great many more research studies, which is wonderful. The problem, in a way, is that sometimes they suggest different things, sometimes contradictory things. They may not be as applicable, that is research studies, if you're finding it may not be as applicable. So we have to really take the totality of what the research evidence suggests. Within the law and in the guidance from the department, it talks a lot about what happens if you've got a Tier 2 intervention study that suggests this is not effective, but a Tier 1 study that shows that the very same thing is effective. And then what if you have another Tier 2 that shows it's ineffective? How do you balance that out? You have multiple studies of less strength, contrary to a single study of stronger strength. So states and localities have to figure out how to balance those things. Number two, the relevance of the evidence to the setting of the population and the need that you're trying to hit. And the door is open to a broader range of research, but there's not been research done on every possible thing. So if you are in an urban school system dealing with a very unique population trying to solve a very unique problem, there may not be research studies that speak directly to that constellation of factors. They have to figure out how to interpret the research that is existing and how it might inform what you're doing within your local setting, the population of students who work with the need, and so on. The third consideration is the capacities to effectively implement the activities. So one of the things that was interesting about the no child behind the error for a random clinical trial is that lots of things were attempted to be studied in the clinical trials, especially curricula. We're finding on the evaluation side that schools and classrooms had a hard time actually implementing the curriculum, for example, of the thing, the way that we thought they were going to do and in a consistent way. Another implementation was hard. There's no surprise anyone on this webinar, watching this webinar as a teacher, actually doing things in the real world is very different than doing them when you're doing a study on them. But these variations or the differences in how where schools and local agencies are enabled in their ability to implement activities come really quickly important. You may find the best things in sliced bread but realize it'll require you to have twice the number of teachers in your school system. So considering that capacity to implement as designed. And finally, the cost-benefits analysis. And really here, I want to think about the finance but also time and opportunity cost and education. Any hour you spend doing one thing isn't already enough to spend doing something else. So it's a real important balancing act between this program might be research-based but require me to spend four hours a day. This other program might be a little bit less impactful in terms of outcomes but I want to spend an hour a day doing that with my students. That creates a cost-benefit analysis that has to come into play. And then there's considerations and there are probably many, many more come into how do we map out the breadth of research that's out there to meet our locally defined or looked identified needs. So I thought it was going to complicate things a little bit. I do apologize for that but I think it's a good time to stop and catch our breath, catch my breath anyway, and also open up for any possible reactions or questions. Well, thank you very much, Kyle, for making this even more complex. I do think that you make a great point, though, that sometimes the tiers can appear cleaner on paper than they do in practice in real schools and classrooms. So I appreciate you pointing out these nuances. I am curious if anyone has encountered some of them in their own districts or state departments. Okay, well, I appreciate you discussing kind of how these factors have to be taken into account when we're looking at even the most rigorous tier of evidence. Oh, thank you. So I mentioned, I'll just segue over to Chris here, the tier four is an unusual piece. And we work in policy and the states really get excited about this and being a really great opportunity. So, Chris, do you want to jump in? Sure. I think one reason why we wanted to focus on tier four a little separately is we don't want you to think of it as a default or just a permission to select any practice or if you're an SEA, a permission to kind of go ahead and approve anything that is presented by an LEA for one of these programs. It really is, I think, going to be an area where it will grow in interest and grow in potential, but only if we pay attention to what it's really trying to do. So it's called in the law demonstrates a rationale, but you've already seen from what Kyle has said, you can't just go by the titles of these things. You can't just look at strong, moderate, or promising and give them your own generic meanings, which frankly I have seen some states do. The definitions really are about research design under those titles. In this case, the title demonstrates a rationale. I want to unpack a little bit. I've seen it referred to as under evaluation or evidence building, and I think those are two really good monikers because they describe a critical thing about this. It's not just about what you might, if you didn't know anything else, but demonstrate a rationale, how you might go about it. It really does have an evaluation component and that's what we want to look at. So the demonstrates a rationale means that the rationale connects to existing research findings, perhaps a study that was done at a different level, maybe something that was done at an elementary level that you're trying to transfer to a high school, maybe something that was done with an English-only population, like a vocabulary building program that you want to try with English language learners, but there is a rationale for why you're selecting this for the need that you have and you're connecting that argument to an existing research base. And that research base has shown that the strategy, the program, the intervention does improve outcomes, and so you have a reason to think in your use it will improve outcomes. And just like the others, those outcomes are either student outcomes or they may be other relevant outcomes, for example, a teacher practice. Now, in addition to demonstrating that rationale, there's also a requirement for evaluating the effects of that program. So already you can see that in many ways, Tier IV represents a higher bar in terms of the amount of work that you would have to do to implement the program. I'm not really simply saying here's the study or set of studies that shows why I've selected this program. Here you're actually doing that plus an evaluation plan. So if you're an LEA or an SEA, you've got these two parts you're demonstrating. Probably the best way, the most efficient way to show the likelihood of achieving outcomes is to develop some type of logic model to present a logic model that expresses the rationale along with the link to the studies and an evaluation plan. It may not at this stage be a Tier I or even a Tier II evaluation plan. It may be a simpler evaluation plan to begin to collect some data about the effects upon implementation. So those two pieces have to go along with it, whether you're promoting it as an LEA or whether as a state you're approving it. So why would you even allow this or why would you even want to go there? Well, you know, we know that in a lot of the areas that people may want to be using resources for, using S or resources for, there simply may not be a reasonably available study. Some of the areas that are of greatest interest, competency-based education, application of technology, it's going to be probably pretty hard to find Tier I, II or III studies that pinpoint positive effects, positive outcomes for exactly the implementation that you would like to use in your own district. As we've already said, Tier I, II, III interventions often have been used with a particular population. Maybe they've been used with a special needs population. You have, you're using it in a broader population, perhaps with high mobility or a different language group or a different grade level. So all of those aspects of the Tier I, II, III studies come into play when you think about how it will be applied in your situation. The other thing, and I think this is going to be very common, is you'll find studies that were very rigorously done, but they're on a very small scale. We've got a lot of literacy studies like that, for example. And the notion of large sample, you're looking at 300 to 350 students in that sample, and many of our well-known studies in peer review journals have been done on smaller samples than that. So you may want to use something that's been well tested, but it doesn't really meet those Tier I, II, III requirements in the way that Kyle laid them out. But I think also, in an important way, you may want to champion innovation and augment this pool of interventions that are backed by some level of evidence because in your state, there may be some really important, consistent patterns of need that people want to test out interventions and strategies that they think will work to ameliorate those needs. So this whole idea of contributing to the research-based building a culture of evidence is really important here. But as I say that, I want to remind you that Tier IV is not allowed for Title I school improvement plans or some of the competitive preferences in grant programs. Some of the places where I think we're going to see Tier IV applied, Title I parent-family engagement, it's going to be a really useful area there, Title II professional development, teacher learning, teacher application of what they've learned in professional development. Title IV, where you might see the use of technology, social-emotional interventions, for example, other changes related to school climate. There's some implications for the state role particularly, and you can, if you're in LEA, you can also think about these implications that this notion of this culture of evidence, you want to promote investment-improven practices, really taking that seriously. And so, you know, I would say I've seen in my career often people telling people they must use research-based interventions and then on the other end of that, in areas where people simply can't find those research-based interventions or they want to argue for what they're interested in using that doesn't have any research, people sort of turn a blind eye to that and simply say, well, you know, it's been written up in an article or they just assume it's research-based. Here, we're really trying to turn the corner on that and actually promote investment in improvement practices. And that's where this Tier IV is kind of a relief valve for that. It does mean that high-quality needs assessments are important. We really want LEAs to be pinpointing needs because we're going to be matching research-based interventions and strategies to those needs. So it calls for paying perhaps more attention to needs assessment than has been done before, not simply filling in a cookie-cutter sheet about sort of traditional data about the school but actually taking a deeper look at needs. And then related to that, it means approving those plans, looking at those plans, looking at the selection of interventions in light of the needs that were identified. That goes back to the grade levels, the populations, the particular context. So that's both more a higher obligation for the part of LEAs as well as SEAs. One thing that you might see is more piloting work done, encouraging piloting, trying things out. I don't have on the slide, but certainly there should be a bullet that says much higher expectation on figuring out how to help districts understand what interventions are available in a different area, help districts weigh the evidence that's there, and basically help people figure out whether there is something that has a proven practice that can meet that fifth grade need for improving students' understanding of fractions, ratios, et cetera. And that's one place where the comprehensive centers come in, the regional labs come in because some state departments have that research capacity and some do not. But if we want to make this real, we're really going to have to help point LEAs to where they're going to find this information. And then finally, states may want to consider how they're going to help invest in evaluations so that putting the burden of carrying out a formal evaluation on districts without any help is a big burden. So being able to use perhaps some set-aside dollars at the state level to help support evaluations in areas that are likely to be important to lots of different districts. And so those would be some ways to consider how you might help to fund this aspect. That's a little bit about Tier 4. Thanks very much, Chris. I really think that your point about how Tier 4 can be really an opportunity to contribute to an evidence base that is always being developed. It's, you know, we don't have all the answers and we never will. And so this is an opportunity for some real applied research and evaluation to be conducted. But you also mentioned that it comes with some opportunity cost in that it requires ongoing evaluation. I'm asking you to speculate here, but do you suppose that that will have the effect of constraining states or districts from really considering implementation of Tier 4 kinds of practices or interventions? You know, I think it could. I think one strategy that states may want to use is to not require, I'll use the term sophisticated evaluations at the outset, but to try to do pilots that have shorter term evaluations that may have a lower bar to begin seeing if there's promise in the effects and then invest as a state in partnership with an LEA and if something does show a great deal of promise in either partnering them up with someone in their own state, perhaps a university, perhaps a regional lab, to be able to carry out an evaluation. Or partner districts that are working together is another option so that you can expand. People are investing in a common strategy, maybe a common data collection, but there are three or four districts that are doing this that have high needs. There's also, you know, in Title IV, for example, there is the option for the SEA to treat those dollars competitively and not simply give them out by formula so that they may want to be investing in a smaller number of grants that have evaluation coming along with those grants on behalf of learning some strategies that then can be disseminated statewide. Thanks, Chris. It's really helpful to be reminded of the various supports and ways of self-organizing that can permit those kinds of investigations. So we've thrown a lot at folks today. Does anybody have any questions or comments or observations? You're welcome to type in the chat box or just speak up. Okay. Well, if there are no questions, Kyle and Chris, do you have any final comments that you'd like to make? I think I would just say it's very early and it feels like ESSA has been around for a while to some of us, but we know on the ground it's still very early. So there's a lot of room to experiment with these ideas as people are starting to move into it. And so while, you know, there's a lot of flexibility in how states and locals spend the ESSA dollars, a lot of different possibilities for allowable uses. This evidence-based, I think, people are just beginning to realize is a theme that runs through all of it. So taking a little bit more time than sometimes we do to both do needs assessments, develop plans, figure out what the right interventions are, and then get some feedback on those plans. It's something that I don't think people have often been used to because the plan writing sometimes happens near the end of the school year, but given that we're just at the beginning of all this, it would be really worthwhile to take the time to take those steps very deliberately. Mm-hmm. Great point. Yeah, I think that's right, Chris, and I think also, you know, this is a very general, sometimes rapid fire introduction to ESSA. And there's a lot of detail and, you know, the materials would be available for people to go back to and for others who weren't actually listening in live. But I think one of the things that struck me, especially towards Dan, Chris, of your comments, is it's almost palpable shift in how we really need to be thinking about evidence and research and how we do education. So instead of just going and taking the scientific research space to look off the shelf and giving it to a school to do, which is maybe one way that you could think about a very prescriptive, limited way of thinking about evidence, we're now opening the door to say, okay, there's a whole bunch of research out there that could inform what you're doing. Oh, and by the way, it'd be really helpful, especially under tier 4, but also regardless of what level of evidence exists, to be paying attention because there might be a couple of really wonderful tier 1 or tier 2 studies that suggest doing this thing. But for whatever reason, it's not giving you the results that you think you're going to get. And because it's almost impossible to have done every research study, every possible combination of things. And so this culture of evidence, I think, is really the exciting piece of it, getting really finally encouraged. The application of research or evidence-based approaches into an active verb for more of a passive verb, and I think in how it's implemented, it's really exciting because as you said, we're very, very early in that process, especially when we're wanting to be actively engaged. So I'm looking forward to continuing this work and this conversation. I would say just one more thing, and it's from a project I worked on a long, long time ago with a number of states. The states every year brought together the LEAs and had them share what they've learned from their own implementations and their own evaluations of particular projects in their sites. And I think we often don't do that with Title I, Title II, Title IV. I would say it was a very exciting annual meeting for those locals and states because they were learning from each other. They were sharing not sophisticated research in most cases, but they were sharing their attempts at outcomes, whether they were pre-post outcomes, whether they were observations, and they began to build up a much stronger sense of inquiry, a much stronger interest in results, a much stronger interest in research in general. Yeah, that's wonderful. That's really exciting to hear. Thanks for sharing that. Well, thank you very much, Kyle and Chris, and I'd like to preview for everyone that we have a follow-on webinar planned. And this one will sort of be launched from what we talked about today to delve into how states are actually helping districts to make these determinations about evidence-based practices for implementation. And we'll feature a couple of tools and processes that they're using, so we'll get much more practical in the follow-on webinar. Please visit us on the web, on Twitter, on YouTube anytime. I also want to let you know that the webinar recording will be posted to our YouTube channel, and you'll also be able to link to it from our website. And then finally, you'll be getting an email shortly that will thank you for your participation, first of all, but then also invite you to complete a very short evaluation survey. You're not required to, but we sure would appreciate your feedback. Thanks very much. And again, Kyle and Chris, thank you. Bye, everyone. Have a good afternoon. Thank you. Bye.