 Good afternoon. Welcome to a close-up Ford School speaker series from the education policy lecture series we have here. Today is my privilege and pleasure to introduce Roland Fryer, who is a colleague and a friend, someone who I knew way back when he was less famous. But I'm glad he has agreed to come and talk about some really interesting work he's doing on the Harlem Children's Zone. Before I forget, I'd like to thank some of the folks that helped organize this. First, thanks to the Ford School for co-sponsoring this. Thanks to Beth Rebar, Tom Avoco, Jill and Katie in the communications office for helping put together this. And to Bonnie Roberts as well. And so Roland is an economist. He is the Robert Barron Professor of Economics and CEO of the Education Innovation Laboratory at Harvard University. His kind of original training and work, for those of you who may know him, was kind of more an economic theory which has lots of funny Greek letters and mathematical expressions. He has moved to some extent, not completely over kind of the applied microeconomics side, looking at a host of interesting education policy programs. He, in prior lives, he served as the Chief Quality Officer of the New York City Department of Education in 0708. And he has a number of awards that are listed here you can read, including the recipient of the Presidential Early Career Award for Scientists and Engineers. The 2009 Time 100, Time Magazine's annual list of the world's most influential people. And most importantly for our case here, Roland is a incredibly bright and articulate and engaging speaker, so I'm sure everyone will enjoy the talk. The standard is that Roland, you can decide how many questions to accept as you go. I think it goes to 5.30, so maybe a few minutes at the end for questions. And with that, I'll let Roland take it away. Alright, thank you. Brian's just being modest when I first got into education research. I asked one of my advisors, I said, you know, I want to think about going to education. I think it's interesting. He says, yeah, you really ought to be like Brian Jacob if you want to be good at this. It is so good to be here. Thanks for coming out. I'm incredibly excited today to talk about some work we've been doing with the Harlem Children's Zone. And you guys, Michigan time is 10 minutes late. Do I get to go to 5.40? As long as people stay. Yeah, oh, I'll do that sometime. I'm going to leave for my flight. But other than that, we can stay and hang out. So look, I think the format I like the best is that you should ask me questions as we go along. I really don't like it when I'm in your chair and someone is talking and I'm confused. And there's a burning question in my head and I can't ask it until 45 minutes ago. And then at my age, I forget it. So, you know, just, you know, yell it out, whatever, we'll get to it. But feel free to ask them as we go along. But let me go ahead and get started. All right, so here's a quick outline of what I want to accomplish between now and 5.30. I'll give you some motivation behind why I think the 97 blocks in Harlem are an incredibly rich laboratory for social science, social scientists to understand some deep questions that we've been thinking about for some time. I'll briefly and very briefly go through the related literature. I'll give you a brief history of the Harlem Children's Zone for some of you who don't know what Jeff Canada and his staff are doing. Then I'll tell you about all the data that we've collected and our econometric framework and our results. And then I'd like to draw a bright line after the results. Okay, I want to be very clear about a few things. I'm going to give you results that I believe in. And then we're going to talk about stuff that I have no clue about. All right, and that's kind of the fun stuff. But I want to be very crystal clear about what I know and what I don't know. The results that come from our identification strategy I believe in. What mechanisms are involved, how you break those down, how do you move forward, how do you take this in scale? I have no clue, all right? I got a little more than a clue, but not much. And so I want to make sure that you understand that is a very speculative discussion. And I'm happy to have it. I'm not shying away from it, but I believe much more in my numbers than I do in what's next. All right, so here's the bottom line because I'm sure at some point I have this great quality that tend to piss people off that you can leave. The Harlem Children's Zone is enormously successful at increasing the academic achievement of the poorest minority students in Harlem. And I don't really have much of a clue why. It feels like, and pardon the analogy, maybe it's extreme, but I feel like we've found a cure for a disease that's been plaguing us. And the cure is something like one apple slice four nights a week for two and a half weeks, broccoli every nine minutes, a really dynamic doctor and nurses that seem to care about you. All right, so I really feel like there's something special going on in the Harlem Children's Zone, but I don't really know what. And the problem with people like Jeff Canada is he likes to do a lot of things to help people. And that's great, but not for statistics. And so I think we have a result here and I want to show it to you, but at the end of the day, I don't know why they are having such amazing games. In the classic academic way, I kind of know what it may, but it's not, but I don't know what it is. So here's the motivation. The racial achievement gap is really, really big. It's about the gaps that rise at age two. You don't really see them earlier than that. That could be because they're not there or it could be because we're not good at testing nine-month-old kids. I think we're pretty good at testing nine-month-old kids. I think there are no gaps there, but that's a different subject. I'm happy to talk about that later. There are about 0.64 standard deviations in math, the gaps are, and 0.40 standard deviations at school entry. Now, Brian told me this was a mixed audience, mixed in the sense of some folks know what a standard deviation is and some folks don't know what a standard deviation is. No problem. You should think of it this way. The average kid gains in the average American public school about 0.08 standard deviations per month. So if I do my math right in math, black kids are about eight months behind in math, about five months behind in English language arts when they enter school. You probably know this from the Nate, but the average black 17-year-old reads at the proficiency level of the average white. 13-year-old in our urban centers, this is nationally representative data, in our urban centers it's much worse. I do a lot of work in D.C. If you go to D.C., if you look at the Nate, 12% of the kids performing math at grade level, 8% are reading at grade level in Washington, D.C. This business is so hard trying to help increase achievement, and every now and then you get a big bonus, a big boost and you feel great about yourself and then you get kicked right down. It happened to me this fall. I was in D.C. with one of our schools because I like to be in our schools. We're doing these financial incentive experiments and I like to be there the first time they give the checks because we take so much flak from all the adults, at least the kids really like it. So I like to be there with the kids when they get their first check. So I was there with the kids when they got their first check and the kids were going crazy. They were so happy about the money and they had earned so much and I was so excited. I said, man, well, I think we're really doing something here and a fight broke out next to me while we were giving out checks. I said, hey man, you got to separate that. I said, well, you guys fighting over? He said, mister, tell him that 38 is bigger than 42. I didn't know what to do. I was like, well, there are many attempts to close achievement gap. Because the kid who thought 38 was bigger than 42 was bigger than 42. There have been many attempts to close achievement gap. There have been early childhood programs, lots of stuff, small schools and smaller classrooms. There have been school choice voucher stuff. People tried to find systematic ways to make better public schools. We've had neighborhood stuff where we've moved people out of their poor neighborhoods and moved them to less poor neighborhoods, all sorts of desegregation. You guys all know this. We've tried a whole lot of things in the past three decades. And I don't think we've had big results. In fact, there's a great book called So Much Reform, I think So Little Success. I think this is a typo. But anyway, it goes through these things. I think this lack of success has played into a pretty rankerous debate about whether schools alone can actually close the achievement gap. And in the run-up to the presidential election, there were two kind of groups that were... There wasn't that much distance between them, but they basically were kind of staking out their claims on this dimension. Some folks said schools are alone or not. That was the equal opportunity, equal education equality project. And others said, you need a lot more than schools. You need a lot of social supports, et cetera. Turns out, Jeff Canada signed both petitions. I don't think we can get any data from that, but it shows that he's a smooth character. What I'm interested in is trying to understand whether or not schools alone can close the achievement gap or whether or not we do need these community investments. Let me be very clear. I am not that interested in the Harlem Children's Zone, per se. I'm not interested in evaluating a particular charter school or a set of charter schools, right? You know, since these results have come out, I've gotten lots of calls from people saying, can you give me some results like Jeff? You think I'm lying. So I'm not interested in that, per se. What we're interested in is the bigger question, which is, you got community investment. Should you invest in the community programs that we all know a lot about? Should you totally invest in schools if you want to close the achievement gap? And it turns out that these 97 blocks in Harlem are a pretty good laboratory of at least trying to start to think about whether or not schools alone are enough or whether or not you need community investments. So when I grew up in my neighborhood, the easiest way to piss someone off was to step on their new shoes. In academia, the easiest way to piss someone off is not have them on your related literature slide. It's true, and it's likely because all the academics I know have bad shoes. So I don't do that. I don't put anyone up on the related literature side. I know a lot of people in this room have written incredibly important papers in this literature. But I don't do that. So here's my very brief related literature. There's a huge literature on the racial achievement gap. Lots of folks have contributed to that. The early childhood literature, there's been a bunch of stuff on school inputs, charter schools, class size, teacher quality, et cetera. A bunch of stuff on neighborhoods, peers. This really, the Harlem Children's Zone is putting all this stuff together. And it's very related to a lot of these literatures that you have contributed to. Okay, here's a brief history of the Harlem Children's Zone. Just in case you haven't seen Jeff on Oprah promoting it. It started in 1970 as New York City's first truancy prevention program. It was called Readland Center for Children and Families. Until the late 1990s, basically Readland was an amalgam of after school programs that in Jeff's words helped a handful of kids escape the neighborhood cycle of violence and poverty but allowed many more to slip through the cracks. So his exact quote is, I was helping them by the tens and losing them by the thousands. Okay. So Jeff decided to create a new organization to focus on changing the whole neighborhood. All right. And that's why he dubbed it the Harlem Children's Zone. The idea was to address all the problems that poor kids were facing from bad apartments to failing schools, violent crime, chronic health problems with a kind of cohesive web of services from birth to college. Okay. HCZ started as a 24 block area. It expanded to 64 blocks in 2004. And now it's a 97 block area in Central Harlem. Okay. Now I know there's a lot of, I noticed a lot of my colleagues here who were superstar statisticians much better than I am. This is not the good part of Harlem. Okay. Don't worry about that. This is the Central Harlem. Right. This is just Harlem. Now here are a lot of programs they have. They have their charter schools which admit by lottery, which we're going to use. They have a bunch of early childhood programs. So they got baby college. Right. It's not for the babies. It's for the parents. Right. So baby college is a nine week parenting program where the parents come and they learn things about how to be a parent. Like putting up whatever, Brian's got like three kids. He's my idol on this stuff. I have zero kids that I know about. So I'd just like to be honest. So there's a socket plug thing that you cover, that you cover the plugs with. Right. Okay. So they teach you to do that stuff. My grandmother didn't know anything about that. She just thought survival of the fittest, if you dumb enough to electrocute yourself, that's just you. They also teach about discipline. One of the most famous series of classes in the baby college is about not spanking your kid and using alternative forms. This is something my grandmother also didn't know about. After you go from baby college at three years old, there's a three year old journey which is a very similar parenting program for parents with three year olds. Then there's the Harlem gyms which is like head start on speed. It has a four to one student to teacher ratio. Kids learn three languages in the Harlem gyms. It's basically a preschool program. Just one thing of note, baby college, your parents can go there as soon as, you know, when they're expecting parents. Okay. And one thing that is pretty unique about the Harlem Children's Zone is you don't just go sign up for these programs. You're actively recruited for them. So he has a whole list of full time employees and volunteers that go door to door in project buildings in laundry mats at the check cash and place on the corner. And when they see someone with a small child, they say, you need to be in baby college. It's free. All right. So in elementary school, he's got a bunch of after school programs. Same thing with middle school programs, high school, college. You see, he's got a bunch of stuff here. Okay. And what's interesting, what's nice about it, oh, that's not good. Okay. Let's hope that doesn't happen again. That was a map of the Harlem Children's Zone and their borders there. And let's go back to what you can imagine. Okay. There are borders here in this map. And what's interesting by the Harlem Children's Zone is if you live inside the borders, you're actively recruited for that list of programs I just showed you. However, there are also three to four charter schools in there and those admit by lottery. All right. So you can see where I'm going with this. There are some people sign up for the lottery. Most of them live in the zone. Some of them get the schools. Others don't. Everyone gets the community programs. Okay. That's the type of variation I'm going to be using to try to identify schools versus communities. Okay. In the end, I'll be able to tell you that it ain't communities alone. I don't know if it's schools or schools interacted with the communities that are the most important. But I'm going to make an argument that it's just schools, but it's an argument. I don't have the data to be able to suppose this. Do you have to put your name in the lottery? You do have to put your name in the lottery. Again, you're actively recruited to do so. And there's no stipulations for the lottery. It's not like some charter schools that say, everyone's invited as long as they sign a parent pledge. Or you can do it if you satisfy the following criteria. In fact, I'm going to show you data, at least on observables, that these kids look just like every other kid in Central Harlem. Their parents might not be. Absolutely. Any lot. I'm going to do two things. Let me get to my, love the question. You're right. I've thought about it. I'm not sure I'm going to satisfy. Okay. What percentage of kids you should get in signed up for? Half, because about 200 come in. They only have 100 slots. Okay. Here's the data. We combined two data sets, one from the Harlem Children's Own Administrative File. So we went to the zone and helped them digitize their administrative files that actually have their lottery winners and losers. Okay. We also have data from Baby College and Harlem gyms of all the kids who have actually gone in. Subsequent to the time I made this slide, we also have data on every single community program, time in and time out for all the kids who have gone through that. As you might imagine, it's taken a few trusty-dusty undergrads to get that data into a working format, but we're working on it. We merged that data with the Department of Education at New York data, where we have achievement data, attendance, et cetera, all the administrative data from 0304 to 0708. Okay. And we have the admin files for mastery, lottery winners and losers to the achievement data. So as long as you're still inside New York City public schools, that's Manhattan and the other boroughs, then you're in our data set. Okay. Test scores are only available for grades 3 through 8. One second. Attendance and promotion data are available for all years. Yes, ma'am. What national data set did you use from the Department of Education? This is the Department of Education in New York City. Okay. But what particular demographic did you use? Did you use that set of data from the Department of Ed to compare with the children's own? Yeah. So in New York City, the New York City DOE, they keep data on every single kid. So it's every child? Every single child. So we have 10 years worth of data, it's five years there, but we have 10 years with the data now and all 1.1 million kids for every year. It's a state data set rather than a national data set. It's a city, New York City Department. Just a city. Yeah. But it's all five boroughs. So if the kids move to Buffalo, we don't have it. And how many children are there? We have about 10 million. 1.1 million in the New York City schools. Yeah. It's great for experimentation. No, it's funny because I was there as the Chief Equality Officer. I hate that title so bad. Chief Equality Officer. But it's a long story why that title came out. It was not my idea. But New York City is so big. I remember sitting in my little cube next to Joe Klein and I'd say, hey, how many schools do you have? And they'd say, you know, that's an interesting question. I was like, no, it's not. I wonder that kid don't know if 38 is bigger than 42. All right. So that's what we have. We have New York City Department of Ed data. And we have our home children's own stuff, and we're going to merge those two. All right. So here's how we did that. We mapped the data using the following algorithm. So we took your last name, your first name and date of birth and matched you to the New York City Department of Education with various abbreviations and alternative spellings, et cetera. Here's the quality of the matches we got. For the Harlem gyms, we got 91.2% of the kids who went through Harlem gyms. We found them in the New York City data at some point. In the kindergarten treatment, we got 92.5. In the control, we got 89.2. In the middle school treatment, we got 90.6. In the control, we got 85.4% of the kids. Again, if you move outside the city, we don't have you. Or if you, you know, if your name is, you know, Caitlyn with a Q in between and we didn't get the Q, then we don't find you either. My former colleague Caroline Hoxby, one of our students, estimated that one can expect about 90% based on natural attrition. We're in that ballpark. And the nice thing about the results I'm going to show you is they're so large that, like, even if we took the control kids and said they, but once we can't find, we're really terrible. I mean, the bounds on these estimates will still, even in the lower bound estimate, you'll see bigger things. Okay, so here's the econometric framework in New York question. We're going to do two statistical strategies. One, we're going to use the lottery. The other, we're going to use IV. Okay? So here are the lotteries. The state dictates that over-subscribed charter schools allocate enrollment offers via random lottery. Lottery winners will form a treatment group and lottery losers will form a control group. So the ITT, the intent-to-treat estimate, is the effect of being offered admissions into the Harlem Children's Schools. Okay? So the stuff I'm going to show you is going to be ITT effects. The intent-to-treat. It's kind of like a lower bound. So this means, right, the effect of being offered admissions. Brian wins the lottery. He doesn't go, I'm still going to count him as a Harlem Children's Own Kid. Okay? Or if he goes for two days and leaves, I'm still going to count him as a Harlem Children's Own Kid. That's the intent-to-treat effect. All right? And we're going to estimate that by having the outcome on the left-hand side for each individual eye. We're going to control for some basic demographic stuff, and we're going to include a variable for whether or not you've been treated. That's the Zi. Okay? And then we're going to estimate the treatment effect. Now, the treatment on the treated, which technically not an upper bound, but it answers a different question, is the effect of actually attending the Harlem Children's Own Charter Schools. This estimate is going to be obtained by basically instrumenting for whether or not you went with your original assignment. So we need something that's correlated with whether or not you go to Harlem Children's Own. Okay? And that's going to be what's correlated with that, the original lottery assignment. Okay? I just want you to notice two things. I don't want to get bogged down the technical details unless you have questions, but one estimate is the effect of being offered admission. That's the I.T.T. The other is the effect of actually going. Treatment on the treated. My question was, how did the students get into the pool that was subject to lottery? So they were actively recruited by the staff of the Harlem Children's Own? That's not a random process. No, no. Well, it's not a random process, but it's an exhaustive process. They go out in these 97 blocks and they knock on the doors. They go to the laundromats, et cetera, and they try to get people to sign up. Now, who signs up is not a random process. That's for sure. Okay, so two things. One, I will show you on observables that they look very similar. And two, we do IVs so that we don't have to actually... So this will help us with the external validity. That would be interesting. I don't know the denominator because I don't know how many people they contacted. Just so the students in the region that are eligible to be in the lottery, what percentage of... I don't know off the top of my head because I don't know how many kids are in the 97 blocks. I think... I don't know. I don't even want to guess. If you send me an email when I get home tonight, I can find the number out for you. Again, I want to underscore that this isn't a problem with any lottery-based analysis. And two, what I like about the Harlem Children's Zone is that they actually actively recruit people who wouldn't sign up on their own. So the two important issues, which you just got at one of them, with lotteries. One, the kindergarten lotteries, which were not sufficiently oversubscribed when we have the data. And two, lotteries are not necessarily externally valid. Okay, who signs up for the lottery could be very different from the treatment effect that would happen to kids who actually didn't sign up. So we're going to compliment the lotteries with an instrumental variable specification. So we're going to employ a very simple IV strategy using the interaction between a student's address and their cohort. Okay, so let me just explain this in words. The identification is going to be driven by two forces. One, a comparison of kids within the zone who were of eligible age in that year relative to others who were not. So the Harlem Children's Zone schools opened in 2004. They only opened, and you were only eligible if you were in kindergarten and in sixth grade. So one of our basic counterfactuals in our IV estimate is to say, okay, let's compare the achievement of the kids who were in sixth grade and eligible that year versus the kids who were in fifth grade and who were ineligible in the zone. Sorry, fifth grade is a bad one because they'll be eligible next year. And who will never be eligible for the Harlem Children's Zone. And the second piece of this is a comparison of kids who are of eligible age who are close to the zone versus those that are not. It's the interaction of those two. Jesus, are all my slides this way? This is not, okay. So what are the threats to this IV strategy? Where could it go wrong? Well, if the instruments are correlated with unobservable determinants of outcomes, then we're in trouble. If high achieving people move into the zone when they have kids who are eligible for the lottery, then we're in trouble. Now, did that happen? I don't know, but I can tell you that you're eligible for the lottery no matter where you live. So moving would not have helped you in any way get your kid into this school. Two, address specific shocks. Now, so if individuals inside the zone receive some sort of positive shot that is not received by other kids inside the zone, right? Oh, sorry, other kids outside the zone or any other cohorts inside the zone. What do I mean by that? If I'm giving this talk and Brian says, oh, Roland, I forgot to tell you, in 2004 I went into the Harlem Children's Zone and I gave lectures about how cool it was to be an economist and that if you really study hard in school, you too can be an economist. I would be screwed, okay? Well, I mean, I'm sure that has no treatment effect, but here, so if you imagine, the biggest assumption I'll make all day is the fact that Brian has a treatment effect on this kid. But if you imagine that Brian had a program that had a big treatment effect and he did it in the precise year that we're measuring and he did it only inside the zone and kicked out kids who are outside the zone, then the IV strategy would be invalid. Just to clarify this, what you're talking about now is just how you're going to estimate the effect of the attendance in the charter school itself as opposed to the effect of the community services and so forth? Yes, this is, what I'm talking about now is how I'm going to measure the effect of the impact of the charter schools and I want to compare these IV estimates with the lottery estimates. Okay, and you're going to, I'm asking because the inside the zone, outside the zone obviously has been influenced how likely they were to receive the recruited for services. Exactly, that's the whole point, right? And what I was going to show you with this quick time decomposer thing is that if you look, if you take a line, it's in the paper, if you have it, if you go towards the border of the zone and you look at the probability that you're going to sign up, that probability is pretty flat and then as you get close to the zone, it starts to go up, even outside the zone. Okay, so, you know, in the same way people have used distance as an instrument, we're using distance times cohort as the instrument. I hope the rest of these slides are here. Let me show you some results. This is a problem because I actually don't have the, this is a real, I did this on a Mac and this is not like working. Thanks for telling me. So, how should we do this? I have a Mac, too. If you have a card, I will PDF it on my Mac and then we can put it on this thing as PDF, right? Sorry for the technical glitch here. Okay, I don't need that anymore. Okay, cool. I just want to show you I had a Mac. I know you make a line, but we got a Mac. So, this is the 116th Street, 125th Street, Mass, and so this is where the zone's middle schools are, right? About 134th Street, 131st Street, and over here. That's where the elementary schools are. The elementary school is away from the middle school, et cetera. And I think that has some interest in our policies about what's going on there. But that's that map. Got everything for you now. Okay, here's the thing I promised before. Now I have no pointer. That's okay. So, here's where the identification is coming from. Okay, so if you are in the previous cohort, so if you were a year older, if you were in first grade when the lottery was for kindergarten, you can't get in. So, you see that the black line is being percent enrolled 100 meters, 1,000 meters from the zone, 500 meters from the zone, et cetera. It's going towards the border, which is a little counterintuitive, but going towards the border, getting closer and closer to the border of the Harlem Children's Zone, what we did was we mapped all the kids in 1,000 meter away from the zone using art GIS, using different points from the zone. And here are the test scores in ELA and minor. So, as you can see, as you get closer to the zone's borders, actually the test scores are falling. Now, here is the 2004 cohort. Okay, so these are kids who are actually eligible. You see that the percent of enrolled is very close to zero until you get close to the zone, then it actually starts to go up. Oh, what does this thing do? It's a little thing, put it in the USB. Oh, okay. It's like, you don't give me something broken. All right, let me see if I can do that. I'm sure that all this is going to break. All right. I'm doing a talk about whether or not communities are important and everyone's having to help me with my own talk. I'm going to keep talking and let's see if we can get this thing to work. So, the identification is coming from this, the difference between these two sides, okay? Various cohorts, kids who are eligible, kids who are not, and distance, distance interacting with cohort. Just wanted to show you that. Okay? So, here are the results. She knew that didn't work. Let's see if I can spend up some more of his time. Okay. I had to call you. So, here you go. Here's fifth grade when the kids, so April 14th of their fifth grade years when this lottery happened. So, I'm showing you the test results in fourth grade and in fifth grade. No treatment has happened yet, okay? This is for math scores. All right? The dashed black line is the average black kid in New York City public schools. All right? The dashed gray line is the average white kid in New York City public schools. Okay? The red line are the lottery losers. The blue line is the lottery winners. Okay? So, now to your point, I still don't have it about whether or not they have motivated parents. I don't have that here. Point taken. If you were to do this in Kip or some other charter school in New York City, they'd be a quart of standard deviation above the city average before they ever started. Okay? At least that's not the case here. So, you see what's happening. Now, these are the ITTs. This is just... Did you actually win the lottery? Okay? So, these... So, for kids who went for two days and then left, they're still in here. This is the ITT. This is the lower bound. All right? There are no controls here. I don't have any free lunch stuff. I ain't controlling them or what kind of car they drive. I'm nothing like that. Okay? So, these are just raw treatment effects. Oops, wrong way. Here is if you look at the actual people who complied with the lottery versus the control complier mean. What that means is these are people who actually were admitted and went. So, this gets closer to the treatment on the treated, the effect of actually going to the Harlem Children's Zone. Okay? Again, I have zero controls here. So, what you see is the same thing, very similar, an increase in sixth grade and an increase in seventh grade and a big bump in eighth grade. Okay? Now, we have these data. This is, which cohort is this? These are the 2005 cohort. We have the data for the other cohorts and it's very, very similar. So, this is not a one-year thing that's happening here. Okay? Now, I just want to pause and look at this for a little bit because at least in my work I'm always constantly staring at, like, 0.2 treatment effects. Okay? And so, you sit there and you say, this has two stars versus one star. This is 0.2 and, you know, this is great. This is, like, 1.2. It's very different. Okay? And so, I remember when the first time that, like, you know, you have these data programs and the data spits out. His laser works. Next time I'm going to study the ratio of DAP and laser. Okay, got it. Thank you. You're going to hurt me. So, first of all, I saw this. First of all, I thought it was a mistake. But then, once we really looked at the numbers and realized that there was actually something here, I had a great time because I have, you know, a few full-time RAs and they're like, you know, former Harvard undergraduates, as I said, went out to them and I said, you know, we really got to think about this, you know, what we're going to do about this gap here because there's, like, a white-black gap going on. And I'm not sure why you guys can't actually achieve. I would start with your culture. I said that and they were like, well, if we went to Harvard, like, just calm down. Relax. Okay, so that's math, and that's great. On ELA, things are more muted. Yeah, sure. So even though these huge treatment effects go out in grades seven and eight, the average for all black children in New York City is kind of staying flat. Yes. So is that true, and it's mechanically true because the people who aren't being treated are actually doing worse. Yeah, but this is, like, 200 kids. So, and there's, like, you know, 500,000 black kids. Even if they weren't, even if they were doing a little bit better, you wouldn't expect them to be gone. It's going down. Yeah. So they're actually the ones who were, you know, were complying and actually didn't get in or doing worse. Yeah, what I should do here, this is the average in New York City. What I should do is put the average in Harlem here. Right. And, you know, I can do that. I don't know what it looks like, but it's not clear to me that, you know, we know middle schools are a place where gaps really start to open up, but it's not clear to me that if you looked at just the poorer place, I mean, you know, this is all black kids and the treatment of control is just basically Harlem kids. Okay. That's a fair point. Totally fair point. I should put it here. Okay. Now, ELA is not as dramatic. However, if I'd shown you this first, you would have left happy. You see the same thing here. All of them, they kind of go down and there's a little bit of a difference here. And basically this difference is about whatever, 0.2, 0.3 standard deviation differences. You see a little bit more in the control compliance. Okay. So the ELA results are not as impressive at all as the math results. Why is that true? I don't know. One thing I do find interesting though is if you look at the elementary school results that I'm going to show you in a minute, those math and ELA are both going and locked step together. They're both just as big. So there are obviously these theories out there that say, by the time they're in sixth grade, it's very tough to move language, relative to math skills, et cetera. This is consistent with that. Whether or not that's true, I don't know. Okay, so let me show you the distribution. Because you might be thinking, well, Harlem Children's Zone is probably just good at taking up the kids who are already motivated and the rest of them really didn't happen. So here you just see the distributions between the winners and the losers. Again, this is back to the ITT. This is just the distribution. In fifth grade, it's very similar. Okay? In seventh grade, what you see is that, and this is this math, this is math. The lower kids, you see a push here. Okay? By seventh grade, you see the distributions start to separate. By eighth grade, you just see basically the pulling apart of these distributions. Okay? I always like it when you can give a paper and pictures instead of, you know, if you squint real hard. So that's great. In ELA, you basically find nothing that looks like that. Very similar in fifth grade. Sixth grade, very similar. Seventh grade, incredibly similar. In eighth grade, you find a little push out. Okay? Why do you find that push out? In eighth grade, I don't know. But when I was going over the results with Jeff Canada, I said, this is curious, man. You got nothing here in this little push out right amongst these kids. And if you know New York City public schools, they rate things in scales of four. One and two aren't proficient in three and four, right? And if you look at it like that, if you discretize it, you realize all of it's coming from two to three. All of it's coming from two to three. I said, Jeff, I don't even know a model that predicts that. He's like, I do. I said, what's your model? He said, I told the kids, if you went from a two to a three, you got to trip to Disneyland. So I have no idea if that's what happened, but he is unapologetic that those kids went to Disneyland. Okay, so let me show you something a little bit different, which is you might be thinking, yeah, Roland, well, those are effects from the state test scores in New York City and they concentrate on the test. They teach to the test and I would argue that with you until the night and all that, but that's fine. But let me just show you something that's not even like for public consumption in some sense. Let me show you something that's just for HCZ kind of internal purposes. So we found out that HCZ gives the Iowa test of basic skills. Okay? Just for their kind of internal, one of the internal tests they give kids. Some people give chapter tests, they do that also, they also give the Iowa test of basic. Okay? So here a straight line is normal growth. So if you're going up, you're doing more than average growth. The dotted line is kind of, obviously the 50% time. Okay? I don't have treatment effects here for you because the control kids were not in schools that gave the Iowa test of basic skills. So all I can show you is how Harlem Children's Zone kids are doing by themselves. You can make up your own mind whether or not this is big or small or I don't know. But I think it's something interesting and I want to show you all the data we got. So you see in math scores, they're gaining. On another test, not the actual state test itself. Very similar. This is the 2005 and 2006 cohort here. Similar thing to a different cohort. In reading scores, they're going up in the 2006 cohort a little bit. This is going up and then it's flat. They're not near the 50% time. So that's something. One should consider that. Whether or not the treatment effects are huge. I don't know again because I don't have the control group but I wanted to show you this. This is some summary statistics for the lottery. Winners versus losers. The point is there are no p-values that are small. Here are the middle school lottery results. Let me just put some numbers into the pictures that I gave you because we all have the kind of... You reduce class size from 24 to 16. You get .22 standard deviations. You have a Teach for America teacher. She gives you .15 standard deviations in math, .03 in reading. We can go on about basically .2. Let's just look at what happens. We take the 2005 cohort. You have about .2 in the ITT. It's about .279 in the treatment on the treated. For 6th grade math score. For 7th grade math score. It goes up even more. By 8th grade math score, these are big things. These are standard deviation units. I haven't divided by anything. I didn't pull one on you. Here's the .733 standard deviations in the ITT. 1.112 in the treatment on the treated. ELA score not as big, but when you get to the end, it's about .239, .363. You can look at the paper. I want to get to more things here. 2006 cohort. This is an old table. We now have these results, but you can see the pool samples, et cetera. It's robust across years. That's what I wanted to be able to show you. Now, let me show you some IV results. Yes. It looks like, based on that, only about two-thirds or three-quarters of the people that won the lottery ended up attending were like the difference. I'm just going from that ITT to the treatment on the treated. Not quite, because the treatment on the treated is like, you know, not completely trivial fraction. I'm just curious, do you have any sense where they were going and that kind of gets at the treatment effects? What is the difference between Harlem Children's Zone and is it the neighborhood schools and some other great charter outside of the zone? For the vast majority, it's just the neighborhood schools in Harlem. All right, so let me show you the IV results. Now, we have to decide what inside versus outside the zone means, and I don't have any good theory on that, so I've shown you multiple ways where you take 800 meters outside the zone, you take 1,600 meters outside the zone, 2,400 meters outside the zone, et cetera. And as you might expect, as you get 2,400 meters outside the zone, you're over in Columbia University in the more fancy places, and so the comparison groups are a little different. Yeah. Does Jeff Canada teach African and African-American history in these schools? Yes, but not more so than a typical charter does. I mean, a typical school does. I think the difference is that, where the difference potentially could be, is that there are a tremendous amount of positive male role models, et cetera, who are there teaching the kids on a day-to-day basis. Because they didn't cooperate into the regular curriculum, they're going to have separate classes. I don't know the answer to that. That's a great question. I don't know the answer. Okay, so you see the lottery estimates in general are big. I mean, the IV estimates in general are bigger. Okay? So now, by eighth grade, you're looking at 1.3, 1.5, 1.3, depending upon how you actually define this. In the LA, you're still around 0.2, 0.3. Okay? So what's kind of... Here's what I take away. It's a little over a standard deviation. Treatment effect in math, about 0.3 in LA. Okay? So we do the results by sub-sample, by girls and boys, for example. And you don't really see big differences, except for the eighth grade math score here. Okay? And you see it on boys. Again, that's something that we haven't, has not been consistent in the literature. You know, it... My reading of the literature is, if you look at intervention projects, the ones that work are more likely to work for girls than they are for boys. Harlem Children's Zone is a little different than that. Okay? The elementary school IV results, okay? Here, I don't have nice pictures to show you, because they were just in third grade. So it would just be a dot. Here, you see the math scores are about 1.92. Now, you're saying, why are the elementary school things larger? They've had an extra year. Okay? So the average, basically, is about, you know, 0.4 a year or so. And these kids had an extra year, because they went kindergarten through fourth grade. Right? These other ones are just sixth through eighth. So they've had an extra year. It's a larger treatment effect. What's interesting here, is that the ELA scores mimic the math scores in the elementary school. Yes, ma'am. Oh, sorry. Sorry. Oh, I'm sorry. No, I was, I saw her. I was just ignoring her. No, I'm sorry. What you want? Oh. Yes. Great question. We can. We just got the data. So we just got about, maybe three weeks to a month ago, we just got individual question level data for 10 million children in New York City. And, um, no, undergraduate girl, that's what you do. So we have, we have that data. And so I'm not sure I'll be any good at subgrouping the questions, but I think the state test actually puts them in subgroups, as I think while I'm talking. And it's like, yes, we can do that. We ought to do that. Very fantastic question. And because of my age, could you email me that question? Thank you. Great. We present lottery results, but as I said, the lotteries weren't really over subscribed, but I just wanted to show you everything I know. The treatment effects are still big, but the standard errors are enormous. Here's the Iowa Test of Basic Skills of the elementary school, 2004 cohort, 2005 cohort. What's interesting about these is now you're starting to get above the national meeting, you know. And so, again, I don't have treatment effects for these, but these are some pretty, well, a little wild, but pretty on average, pretty big gains coming, even in the Iowa Test of Basic Skills. Okay. Now, I want to keep plowing through here. That's all the data I have that I can do IV or lottery estimates on. Now, I really want to show you some results with baby college and with Harlem gyms, even though I can't do it justice. They don't admit in baby college in a way that would allow you. They don't do it by lottery. Everyone. They want everyone to be in baby college, right? I know good for people, bad for me. Or Harlem gyms, they don't do it either. So what I'm going to do now is show you some really, really simple OLS of just what happens to kids who come into baby college versus those who go to Harlem gyms and which way the selection goes. I don't know. Okay. I don't know if people select in who are worse. I don't know if people select in who are more motivated. All I'm going to be able to tell you is basically there's very little evidence of a big treatment effect on baby college. Okay. So here, what I have here is Harlem gyms and baby college results. Now, I'm looking at the math scores in third grade. So I've traced people who have gone through baby college and are now in third grade. And what I'm doing is looking at your third grade scores on the left-hand side, controls on the right-hand side, plus an indicator of whether or not you are in baby college. I'm reporting that indicator here. Okay. I'm also doing the same thing for Harlem gyms. I've also done the same thing for Promise Academy in an OLS way so you can kind of see OLS versus OLS. Okay. Promise Academy still looks big. Harlem gyms. You know, the problem with this is point two, which is basically like the effective head start, point two, but it's measured with such error. I can't tell you much. Okay. So it's, it could be, I don't know. What I'm more confident in is that baby college doesn't seem to have a big effect on test scores. Kids are coming to school more. They have less, sorry, that's the L.A. school. They're total absences. So total absences seem to, this is point four. It's incredibly tiny relative to standard error. So I would say on these dimensions of achievement, I don't see any evidence that baby college had any effect. Okay. Now, when I told Jeff this, I said your baby college ain't nothing. He says, because that's the way I talk, he says, oh, but you got different outcomes. You think baby colleges to increase test scores and total absences, baby college in his vision was to get parents to stop cursing at the kids. So he says, I don't know if that has a treatment effect or not. We have different, different views, different outcomes. Okay. And so one of the things I want to do going forward is to collect more data on what's actually happening when people go through baby college, et cetera. All I can show you is what I have, right? Yes. It's a long list. Do you want me to? So, you know, like, you know, the stress levels, I want to understand the interaction of the parents I have with the kids having come to baby college. I would like to figure out a way to understand, even if it's just the survey data, I want to observe parent-child interactions to figure out if the, whatever they call it, the culture, cultivation, that theory that they teach in baby college is actually sticking. So, Bill Wilson, Orlando Patterson, Rob Sampson, myself, we're putting together a qualitative team who's actually going into the Harlem Children's Zone to actually observe these other outcomes that it's hard to get at through administrative data, but I don't have that. These are only students who ended up in the charter school? No, these are students who have been to New York City public schools. So, who is the, some of the people, there's some subset who went to baby college. Yes. But then there's a million kids in New York City who didn't go to baby college. Yes. But the number of observations here is 109. Yeah. Who's the ... These are people who signed up for the lottery. Okay, so it's only people who signed up for the lottery. I was trying to take out some of the effects that you're probably thinking about. Yes. Didn't get in and went to a public school. Went to a public school, but also went, so these people went to baby college? Went to baby college. Yep. But they're not in the charter school. Not all of them are. So that's why we control who went on to attend the charter school. Have you done this for the students that went into the charter school? Yes. So look at the effect of the baby college? Yes. The samples are too small to actually show you anything. Here's the best evidence I got. Forget the regressions. This is just saying did you go to Harlem gyms or not on your Iowa test of basic skills? Those lines couldn't be more similar. Okay. Again, this is, I want to show everything we have. I'm much less confident, obviously, in these results because there's no identification strategy than I am the others. But I will say if someone asks me is baby college knocking it out of the box, I would say I doubt it. Okay. So again, you just see that these things are following very similarly. Whether or not you went to Harlem gyms. Okay. This is what I'm thinking. Okay. Going back to your, this is the speculative part? Oh. Yeah. Let's call it that. No, this is, I want to be honest, I mean, I don't know. I mean, I'm confident that these regressions are correct. What they measure is a different story. The graph you had up there on performance of in lottery. Yes. In school, not in school, but in the lottery where the performance of the kids who did not get into the school but who were in the lottery was flat or declining. But the performance of the kids who got into the school was increasing. Yeah. That very dramatic results for the math. I'm interpreting that graph right. Yes, that's true. Okay. So all of those kids, those two samples, the ones who performed well, on average, their parents are the same. Yes. Because it's just a random draw, right? That's right. In that school, if that school is doing something for those parents who want to do something and it's working, you might be able to pick that up with the, whether or not they actually went to the to the baby college. Last statement might not be right. No, so I think what you want me to do is an interaction term between baby college and the treatment, whether or not you use the baby college and the treatment. So what you really want, now that I've got to click at the works, if you want, you want this, you want the sub-samples to be baby college and not baby college. That's right. If I have the data, I can do it in minutes. I don't know if I have the data. I'll check. It's a great point. I'll see. You know, the problem with this obviously is, you know, we've got 486, even when we're splitten by gender, we're on the edge. So we have to be a little careful. Okay? And, you know, more years, more data. I mean, I think of this as the first project, not the last project in terms of treatment. Okay. So let me keep rolling here. Okay. What have we learned? And then I want to open it up obviously for discussion. And you can beat all this up more than you already have. I think what we've learned is that the math achievement and ELA achievement is not as good, ELA and math in elementary school and math achievement in middle school. Now what else can we squeeze from the data? Okay. Now, here's where I'm speculating. Is it communities versus our schools? So let me show you what investments you get if you're in community versus school, reiterating what we talked about before. The community bubble, you get the early childhood programs, the public elementary school programs, et cetera, is also a kind of student-family bundle. Maybe it's a bad way of thinking about it, but if you're a sibling of a kid who gets into the lottery and you're in that family, you're going to get some sort of investment. Right? Because Jeff Canada, he says fruits and vegetables home. Okay. He says pre-made meals home to the kids. He gives a bunch of material support, advice to parents. You also have greater knowledge about the community programs if you have a sibling that's coming home from the all-night basketball game that's happening at the rec center. They told me about it today in school. That's what I think of the student-family bundle. Then there's just a pure school bundle. Now, what is Jeff Canada doing? On several trips to the Harlem Children's Zone, I've tried to figure this out because it was a little mystical when you first see it. So we went to the Harlem Children's Zone and we just interviewed the principals, the teachers, the kids, and that kind of stuff. I don't know what I heard because the principals, I would say to the principal, look at these effects in math. What do these things happen? He said, strong principals. I mean, you go to the teachers and he said, how do you think you can explain this? And they say, just good teaching, boys. So I'm not sure we got much out of it, but what we did get out of it was that the kind of list of things that they're actually doing inside the school. So they give health services. So if you're sick, you're not feeling well, you go right, at least in the middle school, the health and dental services are right inside the building. They're not if you're in elementary school and I actually think that's interesting. They have social workers inside the school. So I remember when I was a kid and you look out and see it was sunny outside, you just hit someone upside the head, you can go home. Social workers, so you get in a fight here, you get pulled away, why did you do that? Do you feel regret? All that stuff. And then you go back into the classroom. So they also have data oriented instructions. So it's not like the term, which is used a lot in education reform. What they do is they test kids every six weeks. They break the test down by skill. So they know whether or not Mel is with one unknown. And then he gets two hours after school every day to practice on the things that he's the weakest. Right? Okay. You got more time in school. These kids, they stay in school from eight to six and they only get 27 days out in the summer. In the winter, when they go home for Christmas break, just Canada pays them $50 a day to come back. Okay. So if you actually calculate it, they spend twice as much, literally twice as much time in school relative to kids in normal public schools. Every time I want to borrow money from Jeff, I tell him, I'll tell your elementary school kids, it's not normal to be in school till six. He's like, don't do it, don't do it. This may seem silly, but they really, you know, like I've been in a lot of schools and the environment in the schools is pretty incredible. Like the teachers seem to sound silly, but they really care about these kids. They have skilled teachers. They have student teachers. So he's got like the whole foods basically there. Like, you know, you go in, you get like whole grain and like, you know, broccoli. I never eat there. It's ridiculous. They have a coordinated after school program. So it's not like your normal after school program, which I participated as a kid where you break from school, you get like 10 minutes and you go to the after school people and you just show out, right? Not there. You're working on your stuff. It's very, very different. Now, this is cool and all, but it makes the statistics impossible. Okay. So communities or schools. Well, I don't really know. Okay. But let me tell you four pieces of evidence that suggest that changing communities alone won't do the trick. Number one, our IV strategy compares kids relative to other cohorts in the zone that were not eligible for the lottery. So in some sense our IV strategy purges much of the community effect. Okay. Second, I'm going to show you these results in a minute, I think. Siblings of the zone kids who had access to all the community programs, but for random luck, we're not actually in the in the schools show no gains. Okay. So if my brother gets in the Harlem Children's Zone versus I don't into the actual schools and I don't, I'm still in the community programs. I know I have pretty good knowledge of the community programs because it's my brother. My test scores don't move. My absences go down and that's probably because my brother's in school all day and my mother's like, you need to go to school too. But my achievement doesn't actually go up. Okay. We'll be doing analysis of sub-samples like the boys versus the girls. We notice the kids inside the zone don't have any difference in effect of the actual charter schools relative to kids who were actually outside the zone. Fourth, if you look at, if you look at the MTO experiment, you know, where they moved neighborhoods, I think if you designed an experiment, you said, Roland, we should design an experiment where we're going to change neighborhoods but not change schools. MTO would probably be like really, really good. I mean, like to a first order approximation, that's basically what they did. Okay. I mean, not on purpose, but folks move neighborhoods but they're going to very, very similar schools and what you see is that the girls achievement doesn't change. The achievement actually goes down a little bit. And the last thing, which is purely speculative and it should be a separate point here, is if you talk to Jeff, you say, why did you start charter schools? He said, because my community investments, I didn't see any return in terms of the actual achievement of the kids going up. Okay. So four pieces of evidence, one anecdote. Here are the sibling results. You see that the math score, it's 0.2, but it's measured with such error. ELA score is this. The absences are actually statistically significant, so nine less absences. Being on time to grade level is nothing more. So these are looking at just the set of siblings in the Harlem children's zone. One kid goes to the charter school, the other kid does not. You don't see differences in achievement. Now, here's a very speculative discussion and I'll probably end it here, which is, what's driving the school bundle? Okay. You got all this stuff. Health, mental health, longer days, the chef of the school, all this stuff. What can you parse out from this? Well, it's literally impossible to disentangle what the heck they're doing. I mean, we got, they got like 12 things, right? And just a few schools, all of which are doing all 12 things. And, you know, I want to run the experiment where you test all the different possible combinations of the bundle, but that's two to the 12 experiments and I'm too tired for that. And so, but I can tell you, I don't think just based on other people's research, not my own research, I don't think it's teacher incentives and value added alone. Right? Because there's other folks who are doing teacher incentives tied to value added. They're not getting any results to look at anything like this. I don't think it's social workers alone. I mean, we have programs like turnaround for schools, et cetera, that put the social workers inside the schools. They're not getting any effect of anything close to this. It can't, I don't think student incentives alone, I've been doing a lot to those. I can tell you, I don't have any pretty results, but not of the magnitude that these results are. And, you know, I don't think it's longer school day alone, because again, there have been other people who have done things like lengthening the school day and, sorry, and not getting really great results. So, what could it be? I thought they were cheating. I really did. It turns out we got the question level stuff that I talked about before, and we looked at like excess variation. We ran the Brian, Jacob, and Steve Leavitt algorithm for how to catch cheating teachers, and it turns out they're not cheating, or at least cheating very well. You can't tell the difference between cheating very well and not cheating. So, it could be amazing teachers, though. They have some very, very, very, very, very good teachers. Okay, so, eighth grade math, there's this guy named Mr. Patete, he's an ex-marine. He's a six foot three black guy who, you know, does like math believable, right? And he's a very interesting guy. What I think it is and what economists typically have kind of troubles thinking about is interactions between elements of the bundle. Okay, and so, you know, at least when I was in graduate school, the cool thing to do was to estimate a partial derivative. I held everything else constant and I increased the school day. So, I know exactly the school day, you know, because it wasn't cool to have like four acting in ways you didn't know, but the combined effect was large. That wasn't cool. What was cool was to estimate a partial derivative, right? And my sense is, I'm not saying anything deep here, is that the total derivative may be far more interesting, right? If you have better teachers with a longer school day, with something else, maybe that total, right, you remember the chain rule, maybe that total derivative is actually a lot higher. Okay, so I think trying to understand what's really important, let me tell you about what we're doing next, and then I'm happy to stay as long as you want for questions. We're looking at longer term right now in non-educational outcomes, things like teen pregnancy, crime, and so on, okay? And before I'll make a stronger statement about communities versus schools, I want to look at other outcomes where you think the communities would have much more of an effect than actual just test score achievement, et cetera. The Obama administration, who plans to roll out 20 of these across the country in a way that will, you know, putting them together in a way that will maximize learning, right? So, you know, I'm a nerd and I called up folks in the administration, I said, hey, you're rolling out this thing, how are you rolling it out in a way where we're going to be able to learn which kind of is what's driving what, because you can roll it out in a way, they said, that's a good question. I said, no it's not. It's been a billion dollars, you ought to be able to tell me, you know, how to do that. And I have concrete ideas about how to do that, I'm happy to talk about that. The last thing, and then I'll shut up, is one of the things I'm so interested in is, of all this, is trying to figure out, okay, let me piss some people off, what the pill looks like. Okay, I want to figure out what the magic bullet might look like. You know, he's doing 12 things. Could we figure out four or five of those things to actually try to find a random school-based trial in regular old public schools. Because personally, I'm not that interested in just, you know, there's a set of charters that are all closing achievement gap, that's great. The question is, how can we take that stuff and actually close the gap in public schools. And you know, if, you know, we asked Jeff, so I did, I said, hey Jeff, I want to do an experiment, I want to take four things you're doing, what should those four things do be? He would say, human capital, the human capital piece, you've got to have the good teachers, the great people. Longer school day, data-oriented instruction, and the culture piece. You've got to make sure that everybody in the hallways from the teachers to the janitors thinks that 100% of these kids can go to college. If you don't have that, you're going to fail. So I would like to figure out, I know that's complicated, how you do an experiment like that and all that, I get it. But the goal would be to try to figure out, are there three or four things that we should take from this or a KIPP or an achievement first, put into a school-based randomized trial while we get 100 schools to sign up, 30 of them, get this four-element treatment, the others don't, to see if we can actually get gains over three years that look like the Harlem children's. Last statement there, I promise I'll shut up. Here's why I'm so excited about this. I think before we conducted this analysis, at least for me, I was sitting on my computer and I was like, oh my God, nothing works. Nothing works, nothing works. It's contributing to fatigue, people saying nothing can work. This is all you're just wasting your money, nothing can work, nothing can work. Now I feel like we're in a spot where there is something out there that is actually, it has parts of the the time it's actually working. The question is, how can we boil that down to pill form so you can transport it to other places? But for me, that's a much, much, much, much better place to be in. I know there's something, I just don't know what it is, versus maybe something that is, what it will be. Okay. Yes, ma'am. A quick question about the 20 places. Obviously, the first thing that needs to be part of the pill is somehow cloning Jeff Canada type leadership. I totally disagree with that. In the best way, with all due respect. I just, Okay, the question is, she says, and I get this question all the time, is that one of the most important things is going to be to clone Jeff Canada. No, no, it was his type of leadership. Okay. His type of leadership. Okay. That's a broad concept. Okay. That's a broad concept. So now I'm more on your board, more on board in the sense that he is a good manager. Okay. He does, and that's it. He's a good manager. And we, the supply of good managers, I think is large. So I don't, but a lot of people think he's a mystical figure who somehow miraculously can change schools because he's so smooth. I say, no, he's like my uncle. He just got a job. That's the only difference. And, and he's, so I mean, he's good at this as he can raise money, but he's a good manager. He has good performance management. He sticks to goals, has timelines. And I think we got a lot of folks out there like that. We certainly got 20. Yes. Children don't. Yes. That's a, that's a very important, it feels like they belong. Yeah. I, I can add it to my list of things I don't know, but yeah, I think that, I think you may be right. Yeah. Are you familiar with that experiment that they did back in 84 down in North Carolina state where they had a group of college students who participated in the Afrocentric classes versus the non and the ones who were in Afrocentric classes had their GPAs improved by 1.2? No, I haven't seen that. Okay. But I'm just, Please send it to me. I would like to read it. Okay. Yeah. I think, I think you, you could be right. Yeah. Are you familiar with that experiment that they did back in 84 down in North Carolina State where they had their GPAs improved by 1.2? Yeah. I would like to read it. Okay. Please do. Please do. Per kid costs. Yes and no. I have a, I have a, cost per kid, I have a cost per kid in the zone. I don't know what denominator kids take up the program. Okay. So that's 5,000 dollars per kid. Okay. He raises $50 million a year. He's got 10,066 kids in the zone that he treats for one service and, you know, so that's the 50 million. How much of that gets allocated to the schools versus not? I don't know. But it's, it's in the, it's in the three to five thousand dollar over and beyond what normal New York City public schools get per kid. Yeah. Yeah. Yeah. That community is making good connections. Maybe it's a language now. I'm just wondering about how the children's donor project in Canada would recommend connecting with the community if that's the case. Um, I will admit I really don't understand your question but I'm happy to answer it anyway. Um, um, yes. Okay. The slight of hand I did on you today was I redefined what a school is. Okay. That's the slight of hand I did, which is I've got, you know, I've said it's schools but I've got schools that, uh, got, you know, social workers inside the school, you know, bop, bop, bop, bop, bop. Right? The reason I did that is because I think we, there's a lot of public schools who are doing that, like the number of New York City public schools I've gone in and seen a dentist in the next room from the ELA teacher is amazing. And so I think we can get there in terms of our definition of school. So I think that, I'm using the broadest definition of what school is and I just really in terms of driving achievement. Now whether or not they affect teen pregnancy and those other things, I don't know that yet but that's coming. These kids really aren't old enough, right? The oldest kids in the zone right now are in 10th grade. So they'll be in 11th grade soon and then they'll start doing this stuff. I did 11th grade and we'll get some good data. Okay? But, so that's just the reason, just a time constraint we're waiting. Yes. Oh, it is lower than, it's, I think it's roughly the same. I think it's a little lower than public schools in Harlem but I think we reported in the paper. It doesn't really matter for our statistical analysis because as soon as you are in, we're fine but I don't, we reported in the paper I just, you know, age. I don't know the number but it's in the paper. Yes. We need to talk a little bit more about what he did. Yeah. Because there's not very many people in our country that are doing what he did which is, he carved out his own community and he identified what he thought were over time. Yeah. And he was brave enough to say, I'm going to raise enough money to set my own rule. Yeah. So I'm not going to have to play by the rules of the game as the state or the city or the district or the country dictates it. Yeah. Because, as long as you're a leader that has to be by the terms of how you get your money. Yeah. You can never set it up to necessarily be affected by what you think is a factor. Okay. I was with you until your last sentence which is, he still has to lead by the terms of which he got his money. There's no such thing as free money. I've noticed that. So, you know, I have lots of funders too and they're all like, Roland, we got ideas. Yeah, okay. But I take your point. You've got a lot more flexibility than the average person who's over a school but not the average person who's over a charter school. So two points. I think, you know, the flexibility to be able to develop solutions that you think are important locally is potentially an important thing. Second, which I didn't even mention yet in the talk, is that, you know, if you look at Caroline's work on charter schools, there's a distribution of treatment effects in the right tail of that distribution. Other schools that don't have any of these community programs are getting, you know, exactly the same results but very similar. If you look at the work that Josh Angris and Tom Kane are doing using charter schools in Boston, you know, they're finding very similar results. Again, they don't have the community element or the Jeff Canada part of it. But what they do share is this kind of, you know, a few elements. You know, there's no excuses because whatever. But anyway, so they basically educate kids with whatever it takes. And so it has a lot of the flexibility that you've added. But again, I just, Jeff's wife hates it when I say this. I told her, your man is not all that. She was like, you don't know. So, because I hate to keep harping on that. I just think it's really important because even people who are his funders think, oh, it's just because and this can never be replicated. And I'm like, yeah, but there's lots of there are other schools that are getting similar results. And so, I think that's too convenient to say, this is some miracle. I think the best thing we can do as researchers is collect data and try to demystify actually what's going on and make it more formulated. So, we're going to, that's going to be some of our outcomes that we do. So let me just say, you know, it's, I don't know, I think, I'm not sure I trust people who go around and say, do you want to go to college? Oh, yeah. I don't know if I trust that. Right. But I can tell you, there is no expectations gap between, there are no big group differences and expectations. I think some of the group differences come in and how you actually achieve those expectations in terms of like the systematic ways, you know, I want to be a scientist, but I don't like science, that kind of thing. But I don't think if you look at at least in the data I know about, you look at ECLS, you look at the GSS, et cetera, if you look at those, you know, if you ask, is your kid going to be, what degree are they going to get? Right. An enormous fraction of mothers in the highest poverty places say their kid's going to get a PhD. Okay. So, I mean, again, I don't know what that measures, but it's certainly to the extent that measures real expectations, there doesn't seem to be an expectations gap. What I don't know is how the treatment effect of the charter alters those expectations if it makes them more real, you know, in terms of like getting them on the path to actually get there. So, let me get... So, we should talk about this offline, but like one of the things that I've learned a lot in the last couple of years in terms of trying to get involved in education, realize how hard it was. It's like, one of the things I've learned is like designing a survey question. Quite a bit hard. Okay. So like, even your question, I'm not trying to pick on you, but like, there's no way a kid's going to be like, concrete, I don't know what concrete is. I think it's really hard to design that, and I'm happy like offline after this to talk to you about precisely how you word that, but yet the answer is yes, people have tried to get it there. I don't know whether or not they've gotten it to your satisfaction. Okay. With that, like the thank you all for joining us. Thank you.