 Well, welcome everyone, thank you so much for joining us today. It's a pleasure to be here as part of the conference. I want to thank you for joining what promises to be a fascinating and I think important panel on the learning implications of the COVID school closures and the responses that different education systems are taking. And so this panel is going to address what I think is a really pressing question which is what are the potential long term impacts of COVID-19 on learning and how can education systems respond to those impacts? So we're going to be hearing three presentations today which draw from the RISE program, the Research on Improving Systems of Education program, which is a large scale, multi-country, multi-year research program investigating education systems approaches to improving learning outcomes. We just have 45 minutes so I'm going to jump right in. I'll do brief introductions for speakers and then we'll hear from all three, we'll hear all three presentations and I'm going to save Q&A at the end. Feel free to send questions in the chat directly to me and I'll collate those and pose those at the end of our presentations. So somewhat awkwardly today, I'm wearing two hats, one as chair of the panel and also another as speaker on the panel so I have to introduce myself and then I'll introduce our other speakers. So I'm Michelle Koffenberger. I'm a research fellow with the RISE program with the University of Oxford where I lead research on foundational learning and improving education system coherence for learning. And I'm going to be presenting on work that I've been doing on the potential long term consequences of the on learning of the COVID school closures and possible mitigation strategies that education systems could use. After that we'll hear from Dosey Okoy, a researcher with the RISE Nigeria country research team and also an associate professor at Dalhousie University. He is going to speak on learning loss measures from primary school students in Nigeria and the potential for recovery from those learning losses. And then finally we'll hear from Isabel McDonald, a researcher with the RISE Pakistan team and postdoctoral researcher at the University of California Berkeley and she's going to be speaking about a program in Pakistan working to help children catch up on learning losses by building foundational skills and reaching children at their learning level. So with that we are going to jump right into the content. Let me see if I can still share my screen. Okay, are we good? There we go. Okay, my screen's visible. We're good? I'll take that as a yes. Great, okay, so like I said I'm going to be presenting on work coming out of the RISE program on prioritizing learning when schools reopen and beyond. So there's two things that are well established at this point that I want to use as a jumping off point both for my presentation and also I think for the full panel today. So the first is that learning was already in crisis before COVID-19 hit. This is one of what could be any number of graphs that illustrate this. This comes from the PISA D assessments which assessed learning in low and middle income countries against the SDG goals for proficiency and they found that in many countries fewer than 5% of 15 year olds were reaching basic proficiency even before COVID hit. So the depth of the learning crisis was severe. The second jumping off point is that the COVID-19 school closures have forced more than 1.5 billion children temporarily out of school so the scale is also like nothing we've experienced before. So three questions that I'm going to touch on briefly today. The first is how much learning might be lost in the long term due to school closures? Second, how could these losses be mitigated when children return to school? And then finally, how much of a difference could mitigation measures make for children's long term learning? So on the first, how much learning might be lost in the long term due to school closures? So why am I so worried about the long term learning losses? Why does this need to be a focus in the midst of a current crisis? Well, it's because we have evidence that short term learning loss can continue to accumulate even after children return to school. We have evidence of this from a study of the 2005 Pakistan earthquake where researchers went back four years later and studied children who had been affected by the earthquake and those who had not. And what they found was that affected children were equally likely to be in school. The earthquake had no lasting long term consequences for school enrollment and attendance, but they had much lower learning. So it did have these long term accumulating losses. And what they found was that 14 weeks of missed school resulted in two years of learning loss, just four years later. Why they suspect that happened, I'm going to get into in just a moment. So in this study that I'm sharing on today, we ask a similar question of the COVID school closures. What might these long term consequences be of the current short term learning losses? And to investigate that, I use a pre-existing model of the learning process and model what could happen when this shock is introduced. So a shock is introduced in children's learning progressions and what might that mean for their long term learning outcomes. And so when I model learning loss for today's third graders, if we focus in on say today's third graders, this model suggests that missing a third of a year worth of learning during COVID school closures could reduce these children's long term learning by the time they reached grade 10 by a full year of schooling. Now, when I first ran this model last summer, a third of a year of learning loss seemed maybe a reasonable approximation. Those school closures are obviously lasting much longer. I have since run the model again for half a year and in other work for even longer closures, half of a year's learning loss for today's third graders could produce learning losses by the time they're in grade 10 of a year and a half's worth of learning. And so what's going on? What's the mechanism behind these longer term learning losses? Well, what's happening is that because children re-enter school behind the curriculum, they continue falling further and further behind without catch up in remediation. They missed out on critical foundational skills and they're not able to engage in the new instruction as a result of that. Another key finding from these simulations that we run are that the youngest children stand to lose the most. So now looking at today's first graders, model learning loss for today's first graders suggests that missing a third of a year of learning could reduce their long-term learning by a year and a half. And missing half a year's worth of learning during closures could reduce their long-term learning by the time they're in grade 10 by 2.2 years. And so again, how can this be? How are these relatively shorter short-term losses accumulated into even larger long-term losses? Well, what's happened is children are missing, like I said, these foundations while they're out of school. So if that first grader has missed out on learning double-digit addition, when they re-enter, if they're not given catch-up efforts and they're being taught, say, simple multiplication, they don't have the building blocks necessary to engage in that instruction. So now they've missed addition while they were out of school and now they're in school but they're continuing to miss out on multiplication because they can't engage in the new material. They don't have the necessary building blocks and foundations to do that. On that dire note, what hope is there for mitigation? And I wanna propose that there are things that can be done to mitigate these losses. So how could these losses be mitigated when children return to school? So at Rice, we've been working on a set of principles that can both address learning losses when children return to school from the COVID closures and also that represent similar good practices for what's needed to address the underlying learning crisis. We summarize these as the aligns principles, aligning levels of instruction with goals and the needs of students. And we summarize them in four principles. The first is to set clear learning goals, particularly for catching up on foundational skills. The second is to adapt instruction to children's learning levels, including assessing learning when they return, support teachers and other instructors to implement these practices. And finally to adapt particular approaches to the opportunities and constraints of the context. There's no silver bullet, there's no one size fits all but these principles can be adapted in a variety of ways to meet the context that they're being implemented in. So just super briefly on each of those. So the first set learning goals with a focus on foundational skills. So why the focus on foundations? It's because we have a lot of evidence that children who miss out on foundational skills early on in their schooling rarely catch up. We have evidence of this in India and in Indonesia and many, many other countries where if children start out on flat learning trajectories it's very, very hard to steep in them later on. So I'm worried that young children missing foundational skills may have the largest negative long-term consequences for learning coming out of the COVID education crisis. The second is to assess children's learning levels when schools reopen. Children will be behind but by varied amounts and teachers are going to need a sense of the learning levels in the classroom so they know how to adapt their instruction. So there's multiple ways to do this. One would be simple assessments like the OSRB tool, which is what's pictured here. Another way to do this would be to repurpose existing assessments, give third graders the second grade exam to see what content they've missed out on or lost or other scenarios. All of these options require supporting, equipping and enabling teachers to implement these assessments either in new ways or to implement new assessments. Third, adapt instruction to meet children where they are. So children will need remedial instruction when they return to school. Again, there's multiple ways that this could be done and it depends on context. So one would be to streamline or condense the curriculum by cutting out less necessary content and providing enough time to cover the most critical content and to catch children up. Another way would be to begin the year teaching from the previous year's curriculum so to cover material that children have missed out on. Another would be explicit catch up programs such as teaching at the right level which could group children by their learning level with the goal of moving them up from one level to the next rapidly. Again, these all require supporting, equipping and enabling teachers to implement these practices. The added benefit of these is that these adaptations to instruction can also have implications for long-term instruction as well. So similar practices can address the underlying learning crisis potentially allowing education systems to come back from the crisis stronger in the long run. So for example, curricular reform to better align with children's learning levels and needs could improve their learning in the long term. Improvements to pedagogy such as interactive instruction or other methods that ensure conceptual and procedural mastery rather than just rote learning are good practices that should be implemented anyway. Ensuring children master a topic before moving on to the next so you don't have these situations where children have missed out on necessary foundations and are not able to engage in instruction. Those are good practices that should be implemented to address the underlying crisis as well. These I don't have time to get into but just some examples to say these principles have been implemented successfully in multiple different contexts with different means. So again, the adaptation is key but we have evidence showing that they can successfully improve learning. And then finally, how much of a difference could mitigation measures make for children learning? If we do this mitigation, what could it mean for their long-term learning outcomes? I go back to the model that I started with in the beginning of the presentation and now simulate these mitigation strategies. So looking at today's third graders assuming half a year's worth of learning loss during school closures, no mitigation means about a year and a half's worth of learning loss potentially by the time they're in grade 10. That's the no mitigation scenario that we saw before. Short-term mitigation, doing catch-up programs when they first return to school could reduce those learning losses. And the model suggests perhaps to about 0.9 school years by the time they're in grade 10. Keeping these good practices in place long-term could help systems actually come back stronger than before and our model suggests could enable children to come back from the crisis with higher learning outcomes than if it hadn't happened. So that's the exciting scenario and I think that's something that's worth working towards. It's gonna take a lot of work and planning that should start now while many schools are still closed that has the potential to enable systems to come back stronger. So with that, thank you very much. I will stop sharing and we will go right over to Dosey. Dosey, would you like to go ahead and share your screen? Oh, Dosey, do we have you there? Yes, I'm here just trying to see, is my screen, can everyone see my screen? Yes. Okay, thanks, Michelle, I'll be talking about, I'll be discussing some results of a study we did as part of the RISE project on the impact of school closures on learning and potential for recovery. So the study was organized in conjunction with a group of low-cost private schools in Nigeria. So basically I think the context is the same as discussed already. Basically the contentment measures that were implemented to curb the spread of COVID-19 have severely affected education systems globally. I mean, schools were closed, some examinations were delayed so students couldn't progress from one level to the next. And obviously it also reduced in interactions between students and interactions between students and teachers, all of which could affect learning in the short term and also in the longer term. This is even more important in education systems that were already struggling before the pandemic. So in Nigeria, for example, more than 13 million children were already out of school prior to the pandemic. So it will be interesting, aside from learning, it will be interesting to see the effects of these school closures on sustained enrollment, okay. So the projects, we wanted to answer a few questions. Basically how much learning was lost due to these, due to school closures, these losses permanent or can they be recovered? And what can help with recovering some of these for some of the losses? And then what sort of household characteristics or parental characteristics can help to mitigate these losses? And then the fourth question, which we would not answer in this particular study is about the long-term effects on enrollment. Because what we have found was that as students, basically many of these students were on the margin between going, either continuing in school or dropping out. And with school closures is definitely possible that they basically just dropped out and not returned. So that's a different, that would be a different study. And so the study sample, we focus on a group of low-cost private schools in Nigeria. While low-cost private schools, they currently have a high and increasing share of students of primary school students in Nigeria. And they were less likely, in fact, in our sample, no school received any form of government support, okay, in terms of law, remote learning. And then they, so these are schools that have basically the proprietors and then 304 teachers. So these are very small schools. So they were unable to implement any remote learning measures. And then as I already explained earlier, these are students that are again on the margin between staying in school or dropping out. So we select students in grades one, two, three, oh, sorry, not grades, actually is from grades two to four, not grades one, two, three. Grade one, they're too young and then grade above grade four. Many of the students were preparing at home for examinations to get into high school or secondary school. So we only use students from grades two to four. So that's a typo, okay? So what we do with these schools is to implement supplemental lessons for students that have been out of school, okay? So schools closed in March, 2020 and they were not allowed to reopen until September, 2020. So students were out of school for roughly six months. And then as schools were planning to reopen, we work with some low cost private schools to implement supplemental lessons before schools opened completely. And so basically, and then we basically, we have test results for students prior to school closures basically about a week or two before schools closed. And then we test students immediately at the start of the supplemental lessons just to see where they were at that point in time. And then these students were also retested one and then two months after the supplemental lessons. So these lessons were intended to help students catch up with well, basically to have them catch up with what they might have lost and also have them catch up with where they would have been, okay? If schools were not closed. And obviously the sample of students, the sample was restricted to students who were who returned. The lessons themselves were free to the students, but then so some parents were able and willing to pay. So some parents paid for their children to participate in these lessons. But then there were lots of parents who were not able to pay. And so the issue here is that if parents are not able to pay, the schools are not willing to implement these programs because they don't have any funds to pay teachers. They need, so in a typical school, they will need about 30 to 40 students who are paying in order to and in enough revenue to pay teachers. So I guess that's where we came in. We basically covered fees for students whose parents were not able or willing to pay. Not all students were able to return. Some students had moved away from the area. And then for some students, basically the parents were not interested in having them return to school. So that's an issue. And then we also collected data on schools, okay, schools, they are in sort of outlook what they plan to do on teachers and also on household and parental characteristics. So the empirical framework is a fixed effect model where tracking the same students over time, okay, within a school. So basically the change from period one to two will indicate how much learning was lost for a particular student. And then the change from period two to three will indicate recovery after one month. And then the change from period two to four will indicate recovery after two months. So period two is at the start of supplemental lessons. And then period one is basically the period just before schools were closed. And then we note that the learning losses we estimate are potentially lower than how much learning was actually lost for two reasons. The learning losses we estimate so the estimate is based on the sample of students who actually return to school, okay. So those might actually be students who have some activities at home or whose parents were interested in continuing. And then the estimate is also based on what they knew in March, okay. So I think one might get a higher estimate if they were able to compare these students to students who basically stayed in school from March to September, basically stayed in school all through, all right. So we are assuming that learning stopped in March but you could also think of another world where students continue to learn until September. So in that case, the learning losses would have been, or might have, might be actually greater than what we estimate. And lastly, we look at different. I'm just gonna jump, two minute warning. Okay, yeah, okay. We look at differences by sex and also by paratropical characteristics. So basically I'll show you the results and then we add. So this is, so on the left, we have math, okay, these these scores and the scores and then on the, on the right, we have English. The red line is for females and then the blue line is for males. Basically what you see is that from period one to two, okay, there's a, basically on the same test, okay, on the same test, students are performing about 0.6 standard deviations less, all right. So this is about six months of learning lost, okay. So say about six months of learning, about 0.6 standard deviations. So this is the case for math and English. Obviously, but I guess what the positive news here is that students recover pretty quickly. So after one month, they're above the March levels and then they, I mean, the recovery slows down after two months, but it continues to trend up. So this is the case for math and English. And then in terms of parental education, okay, what we find is that students with educated mothers in blue, okay, they lost less learning, okay. They lost less compared to students whose mothers had no education. So it seems like having an educated mother at home made a difference. But again, once students return to school, everyone recovered, okay, everyone recovered, okay. Even those sensitive, educated mothers can continue to do better, can continue to do better, okay. So same for students with educated fathers, okay. So again, it appears that having educated parents, okay, seem to help, all right. And then we don't really find any big impacts of initial income, okay. It's assumed that if the losses and recovery don't vary by household income. And then when we look at what happens, okay, with households who lost income in incomes due to COVID and those who didn't, okay. Again, we find, we don't really find any differences in learning losses, okay. But we find that those whose incomes decreased as a result of the pandemic actually recovered faster, okay. So this is the case for math and English. So how- Josie, unfortunately we're at time. Do you want to make a closing remark? Excellent, thank you. I'm pretty much done, so this is the closing. Thanks, Josie, for that fascinating set of results. Okay, Isabel, go ahead and take it away. I have to take myself off mute first, all right. Let me share your slides. Great, you can see this? Yes, great. Excellent, so thanks to Michelle and the Rise Program for organizing, as well as my colleagues at Pomona, the World Bank, and Harvard, some of whom I believe are here in the audience. So as I go, feel free to put questions in the chat and they can respond to them. So I think Michelle and Josie made a good case that learning losses from the COVID school closures are a big concern. And as we scan around the universe of potential things to do about it, definitely the targeted instruction type programs stand out, where of course the content is targeted to students' actual learning levels rather than their assigned grades. We have some good evidence, mostly from the teaching at the right level program in India, that these programs can really help students catch up and could be a great solution for systems post COVID. But a big question, as we think about implementing these programs at the systems level, which is really core to Rise, is whether these programs can be successful in the regular classroom. So many of the prior successful programs have relied on external volunteers or extra teachers, which can be a problem in a place like Pakistan that doesn't necessarily have that big core of volunteers at the ready. So can teachers in the regular classroom implement these programs successfully? And another big question for us is whether technology can help shoulder some of the administrative burden of these programs and help teachers implement them more successfully. So with those two questions kind of in mind, I will introduce you today to a targeted instruction program that we have been designing specifically for Pakistan and tell you about the research study that we are in the process of implementing this ball in the province of KP. So our targeted instruction program, which we call TIP, is designed for students in grades one through five in Pakistan, really to establish these core foundational skills in math, English, and Urdu. It's designed as a direct response to the COVID school closures and we have been developing it very closely with our government counterparts to ensure that all of our programmatic material align with their priorities. So when I talk about lesson plans, for instance, every lesson plan in our program is directly mapped to student learning outcomes in the national and provincial curriculum, really to be sure that we have alignment with priorities. Particularly these priorities around foundational skills, Michelle gave an excellent explanation of why establishing these foundational numeracy and literacy skills are so critical for every year that a student continues on in school for them to really make the best use of that. And a last priority as we think about implementing TI at the systems level is to think about minimizing costs, particularly around technology. So all the new tech that we talk about in our program is designed to run on existing devices, mainly the personal smartphones and tablets of teachers. So we are avoiding some of the costly tech inputs that might be a barrier to scaling more broadly. So how does this program work? We start out exactly as Michelle said with diagnosing student learning using a set of assessments in math, English, and Urdu. We then use these assessments to sort kids into peer learning groups. So for part of the day, they'll be in their regular classes and then lessons will be shortened slightly. So for part of the day, they can be reshuffled into their peer learning groups. Teachers will then deliver a series of 40 activity-based lesson plans and every few lesson plans there will be quizzes to see how students are learning, which then inform content for these revision sessions that we interspersed. So teachers can deliver some additional activities and content for kids that need a little extra help. Throughout this entire program, we have a tech tool that runs off personal smartphones or teachers can kind of share smartphones and it allows teachers to do everything from see the lesson plans and quizzes as well as training videos on how to teach specific topics. They can also enter in all the grades from the diagnostic test and the quizzes, which first help them do the sorting into peer learning groups, which is quite tedious to do manually, but it also gives them these really nice analytics views where they can see learning across the classroom overall and also look at how individual students are progressing. We have tested the tech tool, the lesson plans and the assessments in a chain of private schools earlier this year, and we had quite promising results. And so what you're seeing here is average learning from baseline to midline to end line for the first three levels of their remedial curriculum across the three subjects. Huge caveat is that we didn't have a control group here and this was a period of high volatility where there were kind of intermittent school closures. We really don't know how students would have done outside of this program. So as economists, we would strongly caution against interpreting these results causally, but was nonetheless quite reassuring to see this kind of learning growth. So we do have plans for a full causal study and RCT that we will be rolling out this fall in two provinces of KP. The treated schools will implement two cycles of the TIP program that I explained and we are most interested in learning, of course, as demonstrated by child test scores, as well as take up of our technology tool that I will talk a little more about. So our first big education policy question is whether this program can be successful in the classroom and how the tech tool might enable that. So we'll have a treatment group that implements the program in an entirely paper-based way and we can compare that to a group that has the technology and then a pure control group that doesn't implement the program at all. And then another set of questions that I think is very relevant for education technology but also probably tech policy more broadly is whether systems should mandate the use of technology or allow people some optionality of whether they want to use it or not. We've heard from some teachers, especially older teachers, that learning a new technology tool can be so tedious and difficult for them that perhaps they might be better off just not using it. So we have some treatment groups where one, teachers are required to use the tech tool. Two, teachers are given full optionality if they want to use the tech tool or implement TIP in a purely paper-based way. And then finally, a third group where teachers have an initial trial period where they're required to use tech and kind of get them over that learning hurdle and then they move into being able to decide whether to use it or not. So comparing across all these groups will give us a lot of interesting insights about the best way to introduce new technology into systems. So this is what we have planned for our first study this fall. We've also got a ton of interest from other public school systems in Pakistan as well as private school systems. And we're really excited about using this as an opportunity to learn more about targeted instruction and kind of unpack this huge bundle of different elements of the program. So we'll be able to look at some different questions such as the importance of sorting, in-person training and then ongoing mentoring and monitoring to try and piece out the separate impacts of that. So I'll stop there. Very interested to hear your questions and comments. Great, thanks, Isabel. I was just about to give you your time warning. So nice work on the time there. Great, well, with that, I wanna start by saying thanks to Isabel and Dozy both for their fascinating presentations. I don't think I gave enough credit at the beginning but they're both presenting hot off the presses, brand new evidence and results coming out of countries that are dealing with these problems as we speak. So I think, just really great to have this new insight to build our knowledge in what is being done and what needs to be done. So we have about 10 minutes for Q&A as planned and hoped. So please, there's a Q&A tab over to the right where you can pose your questions and then I'll feed them out to different speakers. There's two there now, so I'll start from those and then I have a couple of questions but I would rather have questions from the audience. So please feel free to fill that chat and we'll get some questions answered. So I'll start off, there were a couple of questions while I was speaking. One on the learning loss estimates and do those take into account depreciation of existing knowledge? And that's a great question because we have two factors going on, right? It's learning that children have missed out on and not gained because instruction hasn't been as effective remote and things like that. And then there's also actual learning regression like what we saw in Dozy's presentation with these kind of devastating downward trajectories of children losing learning. The estimates I did have a bit of that depreciation built in, not much. And I think as more empirics come back as children return to school, there would actually need to be a lot more of that built in. So I mean, my estimates I presented are actually probably optimistic, which is like hard to see what those learning losses could turn into. But yeah, I think the estimates I showed are probably optimistic because they don't build in very much of that depreciation that we are now starting to see evidence of taking place. And then the second one is what's your take on the demand from governments to verify and act on those losses? Getting back to quasi-normal seems to be the current priority doing new stuff less so. Yeah, I think it's a real mixed bag, right? I think there's a lot of variation across countries. Some countries are prioritizing obviously getting kids back in to take exams. So they're prioritizing kind of pre-exam grade levels to come back. We do see some movement though. There's some work in Botswana right now to implement teaching at the right level type approaches to do catch up in remediation. We see new work happening in Kenya on how to adapt like structured pedagogy type programs to deal with learning losses and COVID school closures. So I think there are test cases where this is starting to be taken up and work is starting to be done. I do think there's a lot of variation though. So obviously there's a lot of places where that's not the case. But there are some that we can start to look to. So with that, I have a question. I'd love to pose to Dosey. Dosey, but I do want to say to the audience, please do add things to the Q&A if you have any questions on what's been presented. Dosey, I did just find that those graphs are... Just so fascinating but also just kind of devastating, right? To see those really steep downward trajectories in children's learning on the same test. I thought that was really kind of a contribution that your study has because a lot of children are coming in and being tested maybe on something else, but it's the same test. So we can see that children have actually regressed or lost learning, not just not learned as much as they would have otherwise. I just wondered if you could reflect on that. So not so much a question as much as an opportunity to say, was that surprising to you and how do you think that impacts how we should be thinking about mitigation? If we have children coming back into a new grade level and they've actually lost learning from the year before, how do you think about those pretty severe learning regressions? Good question. We expected to see some learning losses because usually when students live for summer breaks or something, they lose some. But the estimates, the numbers we have are kind of six times what people have found for regular, you know, you live for summer, you come back and you have a kind of lagging behind. So it was quite a bit more than we expected. And especially given that this is very basic, this is a student grade two to four. So this is basic stuff. So it's troubling on one hand because there are lots of students. In fact, for most students they wouldn't have an opportunity to go through these supplemental lessons to have them catch up. So I think that this is why we wanted to get this paper out just to say that, yes, the students might fall behind but if you put in some effort, maybe in two, three months to get them back up then it doesn't have to be permanent. And then I guess the positive side is that when schools invest in two or three months to help students catch up, they are able to catch back up. So it's kind of, it was, the learning lessons are devastating but then there's also some positive news in terms of what schools can do to, what school systems can do to catch up. So, yeah. Great, thanks Dosey. Okay, so we have a question for Dosey in the chat that I'll read out and then Isabel I have a question for you as well. So I'll pose both of those and then hand it back over to both of you. So Dosey, the question from the chat for you is, was there a difference in learning losses between male and female? So maybe if you could talk a little bit about the gender dynamics. And then Isabel, a question that I have for you is, I'd be interested to hear a bit more about your perception on teachers' responses that you've seen so far. You mentioned older teachers, maybe not being so enthusiastic about it which maybe isn't so surprising. What's the response been from other teachers or younger teachers? Do they see this as something that is, oh great, I'll try this out or are you seeing any interest in maybe this is something that they're interested in implementing on a more long-term basis? Do you see it potentially, are they interested in taking these on in terms of longer term pedagogical practices or things like that? So Dosey, how about you? And then we'll go to Isabel. Okay, yeah. That's a very good question. And it's something we investigated. Mostly we don't find any significant differences in learning losses for males and for boys and girls. Girls tended to lose slightly less in English, in English, but then in form, for math is basically a uniform drop. Girls lost slightly less in English. It's quantitatively about 5.05 standard deviations. But the, it's out of the point, it was not statistically significant. Great, so Michelle, I think maybe I can apply your question first to a targeted instruction program generally and then to our technology tool. So just in terms of targeted instruction more generally, I think the overwhelming sense among teachers and systems is that business as usual is just not gonna cut it given the situation that we're in, which in some ways is helpful because it gives a lot of energy and momentum to try something that's pretty radically different, which the targeted instruction program really is. I think a big concern that we hear from teachers is that they are concerned about whether students will be very discouraged if they get sorted into a group that is far below their assigned grade. But what's actually interesting, what we found in the pilot program is that students quite enjoy the program when they get sorted into different classes because they get to meet a bunch of new students and that's just sort of fun and different and exciting. And then just continuing to emphasize to teachers that the best thing that we can do is put students in the peer learning group that is really aligned to their learning level and make sure that they get the content that they need. And so to try and avoid any sense of discouragement as much as possible. So that's one thing just regard to the program overall. And then I think with regard to the tech tool, there is a difference in how teachers approach the tech tool. Of some find it really interesting and like to click around a bunch and see the different visuals and get very excited about it. And then for some who are just less tech savvy, it does feel like more of a burden. And I think that is what really motivates our desire to try and investigate this really rigorously to understand this heterogeneity in how people approach the tech tool, but then also in the potential benefit that they might have from it. But then also knowing that some teachers are going to be a little hesitant, but in the end, the tool might be useful. That's why we're really trying this treatment group where teachers have to try it for a certain trial period. And then perhaps we can think of technology as an experience good where they learn about it over time. And then they might find that they have learned something about the potential benefit that they might gain. So we'll be very excited and interested to see what sort of results we get. Great, thanks. And I think that brings out an important point too that relates to our Alliance work that I presented on, which is the idea of teaching children at their level. There are concerns about it feeling disempowering for the children to be sorted into lower groups or things like that. But the evidence that we have suggests it's actually empowering for children because what's disempowering is to sit in a class and be completely lost in instructions over your head. It's actually more empowering to sit with a teacher who's teaching you where you are, where you can be engaged in the instruction and be learning. So I think that's a really interesting parallel to things that we've been seeing as well. Great to hear that. On that note, we're at time. So I'm gonna say thank you so much for our great speakers. Really fascinating new evidence. Thank you for being willing to share things that are new and still works in progress. I think it's really helpful for people to be hearing these things and findings coming out of the field at these stages while there are a lot of education systems thinking and trying to figure out how to deal with learning losses. So thank you. Thank you to all of our participants for joining us for this. And with that, I will close out our session. Thanks so much.