 Yes, we have two more great talks coming up. And to the high schoolers out there, I hope you're thinking about majoring in economics right now, yes? And in fact, I hope you're thinking about majoring in economics at Gustavus. If you do, come and find me, yes? I have lots of inside information for you, so. Yeah, great, great. Now, so you spend a day and a half thinking about econ, talking econ, so you probably know that economists are a funny bunch, quite entertaining, aren't we? Dismal science, yes? We mentioned that a lot. We also have managed to predict eight out of the last five recessions, yeah. And another good one is about the economists who, when the four of us go into the room, we usually end up living with five different opinions. Yeah, you might have heard it before. So despite this often cited caricature, we actually agree on quite a lot. I hope that something that comes out of the talks, yes? There is a lot in our profession that despite the many disagreements you may see on the outside, we actually deeply believe, yes? So we do believe in economic growth. Small changes in economic growth have huge impact on the standard of living. We also believe, for the most part, so most of us believe, in open borders and easy immigration and the flow of labor. Another one that we mostly agree on, not all, but most of us, and one that I all like to bring up because it's very counterintuitive, is minimum wage. Do you think, is minimum wage good for unskilled workers? What do you guys think? Does it help unskilled workers? Yes, I expected that answer. Well, economists usually disagree as we worry that if you make labor more expensive, companies will start switching to capital. So just think about it. When next time you're at your grocery store, you're checking out and there's the automated checkout, think about the minimum wage, yes? Anyway, why am I talking about this? Well, there is a reason, yes? So what I wanna talk about is really the microeconomic theory and those are the agreements that I mentioned. They come out of the basic economic principles that my students know that I basically spent 90% of time talking about, yes? And yet in the, so despite the agreements about the microeconomics and the basic economic principles in recent decades, it became very fashionable to kind of dismiss economic theory. And it especially became fashionable as empirical research became easier. So we got very excited about our ability to analyze large data sets. And some people went very far as far as saying, you know, everything economists thought so far is wrong. Throw it out of the window and just do empirical work. So there are people who think that, and there is appeal to the strategy because you know, it's way easier to be original if you say, you know, everyone before me was wrong and I'm the first one here who's right. So it's a good strategy sometimes. And some people adopt it, but turns out you can have a very innovative career, very inspiring career, one that focuses very much on empirical work while also keeping the economic theory in mind, yes? That's where John List enters, yes? So looking at John List's career, it turns out that yes, you can have a spectacular academic career and at the same time be recognized for innovative empirical work and do all this without dismissing the basic economic principles. So as you will see soon, John List's innovative use of field experiments has him the name of a trailblazer and a revolutionary. You're about to see for yourselves that there is no questions as to the originality and the breadth of John's work. And yet for a revolutionary, John List is rather cautious. I guess it's the originality that allows him to stand firm on the economic principles and it frees him from the need to make overblown conclusions from estimating the impact of his findings. So with paper titles such as One Swallow Doesn't Make a Summer, List teaches us to be careful about interpreting empirical evidence. List also highlights the importance of replication, an aspect of empirical research that is very underappreciated, yet incredibly important. Also, List recognizes that first and foremost, economics is an intellectual pursuit with applications to policy but only limited applications. So yes, we can inform economic policy but we can't predict or control human behavior, far from it. So one could say John List is an economist who found the balance. Of course, you wouldn't know this if you looked at his CVS. He's published close to 200 papers, far from balance. He's clearly an overachiever. But with an interesting background, he has born a tracker son in rural Wisconsin. Wisconsin, he attended public schools. He went to University of Wyoming at Stevens Point for his PhD. And yet, okay, yeah, I can hear that, great. And today, he's a distinguished economics professor at the University of Chicago, which is ranked as a top economics program in the world. So what is behind this tremendous contribution to economic scholarship and this incredible success? At Gustavus, we know that there can only be one recipe. It's what we want our students to take away. It's about being mission-driven. Everything else follows. So ladies and gentlemen, please join me in welcoming an economist in balance, John List. Very nice, thank you very much. Very nice, very nice, thank you. Ray Durentis, 14 years old. Antonio Fenner, 16 years old. Oscar Marquez, 17 years old. All of these young men were shot in the streets of Chicago and killed. You see, we have a problem in Chicago amongst Chicago public school students one of them is shot every day. What about the other sex? We face this problem. When you look at the data, what you have here are across several OECD countries. I have the rate of births per 1,000 women, age 15 to 19. I was just over in the UK and they were telling me about the problems with teen pregnancy and teen births that they had. They measure at about 20 per 1,000 women. You look at Canada and New Zealand, they are also arguing that they have a fundamental problem with teen pregnancies. But when you look here in the States, it's off the charts. It's fundamentally different than any other developed country in the world. Now, if you want comparison groups for the US, you need to go to countries like Rwanda and the Sudan. In Chicago, by the time African-American and Hispanic gills reach their 19th birthdays, over half of them have been pregnant at least once. That's a problem that we have amongst the women in America and in particular in Chicago. Now, when you think about what links these two issues, first of all, we know very little about their causes and less about useful remedies. Secondly, you can actually use economics and field experiments, which I'll talk a lot about today, to solve these types of problems. So let me give you some examples. When you think about teen births, you can write down a model. Economics helps. You can write down a model that is predictable. It does good on prediction of who will end up getting pregnant in the community. That model has predictive accuracy. Now, you can say, once you learn about the at-risk girls, how can we use field experiments or behavioral economics as well to tackle that problem? We've done that in Chicago. What we've done is we've given them incentives to be optimistic about their futures. And you know what happens? Teen pregnancies and teen births go dramatically down. They respond to economic incentives in a very predictable way. What about teen shootings? Again, you can create an economic model that can predict the most at-risk kids in our society. After we do that, we can target dollars. And what we found is that the same incentives that work for teen pregnancies, they don't work here. Those types of pecuniary incentives actually do not work. What these boys need is an entire new family system. They need an entire new mentor coming into their lives and making their world a better place. So we have effectively used economics. We've effectively used field experiments to try to make the world a better place when it comes to teen pregnancies and teen shootings. Now, a big problem for both sexes is public education. When you think about how have economists tried to take on this problem of public education, here's kind of the way we think about it as social scientists. We have an idea. We race back to our office and write down a model. We then download mounds and mounds of data. We beat the data up because we wanna say something causal about relationships of the variables in those data. Now, when we've done that, and this goes all the way back to the Coleman Report in 1966, when we've done that, three bits of conventional wisdom have popped up. The first one is we need teachers with higher degrees. The second one is we need classrooms that have lower student teacher ratios. The third one is we need to throw money at the problem. These are three bits of conventional wisdom that when we beat up mounds and mounds of data as social scientists, this is what pops out. So let's now take a look and evaluate how America's done on these inputs. What I have here is a time series plot from 1961 until 2006 that shows you the fraction of teachers with a master's degree or higher. So conventional wisdom has told us policy makers, you need to have teachers in public schools with higher degrees. We've done that. When you look at 1961, you have roughly 23.5%, and you go all the way up here to 2006. Now we have over six out of 10 teachers with master's degrees or higher in public education. So check, we've done that bit of conventional wisdom. What about student to teacher ratios? Over the exact same time period, 1970 until recently, we've gone from about 22 students per teacher all the way down to 15.6. Check, we've done that. What about money? Over the exact same time period, I'm giving you real dollars here. So what that means is inflation adjusted. So these dollars, you can compare Apple to Apple across years. We spent about $5,200 per student in 1970. Now we spend about 12,000 per student. So check, we've done all three of these major changes to public education. Now you can ask yourself, what have we gotten for our money? One thing that people really care about are dropout rates. Let's look at dropout rates over that exact same period. 1970, it was about 76.9% completion. Today it's about 74.7%. Dropout rates are a little bit worse in fact, but let's just say they're the same. They've been flat over the same time period. Some people say, well, wait a second, John, it's not dropout rates that we're after. We're actually after standardized test scores. We care about value added, standardized test scores. I've been cherry-picked here, folks. 9, 13, 17-year-olds here reading and math achievement, looking at a piece of test, looking at international comparison tests. We're completely flat over that exact same period. Completely flat. Now if you're a pessimist, what you will say is, wow, we've really wasted a lot of money. But if you're an optimist, you might say, wow, think about what the world would have looked like had we not done all those great things for teachers and money and student to teacher ratios. The fact is, we fundamentally don't know. This is called the counterfactual problem in the sciences. We don't know what the world would have looked like had we not done those investments. Now in a way, public education is a lot like one of my favorite Mark Twain quotes. All you need in this life is ignorance and confidence. Then success is sure. Now this is exactly what we have in public education. We've been at this for a long time, but we still know very little about how to fix public education. That's fundamentally true. We still have policy makers, whether they're local policy makers or federal policy makers who are like Twain. They're confident, they're ignorant. Now their success is 100% sure. Now what I'm proposing here is to understand the education production function. We have to use our schools not just to teach our kids, but to teach ourselves what works and why. That is fundamentally the only way if we start to learn by doing and use field experiments across districts, call it districts of innovation, where we have several districts at one time doing field experiments to try to figure out what is the causal relationship in getting kids to do better in school. Too many kids are left on the sidelines today because we fundamentally do not understand what are the best ways to do public education. We haven't understood it because we fundamentally misused our classrooms. We haven't used them to learn ourselves. So what's the empirical alternative? Well, I've alluded to it. It's instead of sitting around as a passive observer, waiting for data to come to us and beat it up, let's be proactive and use the world or life as a lab. Let's be proactive and go out and generate our own data in a systematic way so we can fundamentally understand what we need to do to make the world a better place. That's what I'm proposing. Now, some of you might be thinking, well, John, I hear about your field experiments. Who are your subjects when you do these field experiments? So now I'm gonna ask for a little bit of audience participation. I want you to raise your hand if you have flown United Airlines in the past few years. Please raise your hand. Fair number, fair number. Please raise your hand if you voted in either of the past two presidential elections. Wow, a ton of people. My research would suggest that about half of you are lying. But we'll get back later to about why people vote. We'll get back to that in a moment. How many of you have taken an Uber in the past few months? Please raise your hand. Fair number, fair number. Lastly, how many of you, this should get hopefully the rest of you. How many of you have used Google to search for retail purposes? Say in the last five years, please raise your hand. Okay, good, good. Has anyone not raised their hand at least once? Wow, okay. So my therapist is right through those doors. You might need a little bit of help and she'll take good care of you, okay? All of you are my lab rats. All of you have helped me learn about the economic science. All of you have helped me try to make the world a better place by getting people out to vote or trying to figure out why people vote. Try to figure out why people give to charity, why people discriminate. All of you have helped me. Now, I don't know your names and what you did, but I know you as a group of people how you respond to incentives. And that helps me learn about what incentives are gonna do in the future when we change them, how you're gonna react to them. I know exactly how you're gonna react when I change the price on a flight from Minneapolis to Denver. I don't know if you in particular will, but I know as a group, on average, how you will react. Okay, so it's not too freaky, but it's a little bit freaky. Okay, that's what I do for a living. I go out to the real world and I run experiments to try to learn about the economic science. When I do this with governments, I do this with firms, I do this with charities, I do for-profit, non-profit, I do it for academic purposes, okay? So now let's talk about what I've done in education. So this is a map of Chicago and what I'm giving you here is an outline of the Chicago boundary. Now, what's interesting about Chicago is that everyone likes to say that Chicago is a great city of diversity. And if you look at the statistics, it is. This gives you a little different look. What I have here is every green dot is 100 white people. Every blue dot is 100 African American people, et cetera. When you look at a map like this now, it looks like we're a great city of segregated diversity. And that's like a lot of our urban centers. That's exactly what you have in terms of allocation of people across urban, big, large urban areas. Now my lab will be just south of Chicago, a city called Chicago Heights. This will be a school district that has opened up to me about a decade ago and allows me to use it as my experimental lab. And I'm gonna tell you a little bit about those experiments today. Okay, now you can ask, well, why Chicago Heights? Why look at urban centers? This is fundamentally where our problem is in public education. My two high schools, we have less than a 50% high school graduation rate. What I'm talking about here is if 1,000 kids enter the ninth grade, about 470 of them will leave with a high school diploma. 530 will drop out. And they will typically drop out between the ninth and the 10th grades. And that's very similar to other urban centers. When my ninth graders come into the high school, they're reading at roughly a fourth grade level. It makes it very difficult to move these students a great distance and where they need to be when they are already so far behind. So our work, what we've done is we've done a battery of behavioral experiments or field experiments to explore the education production function. Now when I say that, what I mean is what inputs lead to good outcomes? And outcomes we care about is graduating high school, going to college, getting a good job, high standardized test scores, et cetera, et cetera. And we think about inputs as how hard the child tries, how hard the teacher tries, how hard the parents try. Okay, now what I want to focus on here is first let's start with teacher performance pay. All of us know if you look at the popular press, this has become a big hot button issue. So let's see what the press says. First of all, teacher performance pay alone does not raise student test scores. The long failed history of merit pay in how the Ed Department ignores it. Why teacher performance pay won't work. $75 million teacher pay initiative did not improve achievement. One of my favorites, in education reform without merit, performance pay doesn't work, so let's stop wasting money. New York Daily News. But this is actually my favorite one. Here's what teachers unions tell me. John, teachers will never respond to merit pay incentives, they value the kids, not the money. Now when I heard this, this is a conference about the Nobel Prize, I was thinking, wow, 10% of the population doesn't care about money? Where's my Nobel Prize? That's a huge discovery. This is it, John, this is exactly it. This is my time. 10% of the population just care about kids. I could actually pay them zero, I guess. And I'd be fine, or a subsistence level wage, and everything would be fine. But then later that year, the last time they said it, later that year I learned this. Teachers union endorses bonus plan not based on performance. So now it's not, they like money, so there goes my Nobel Prize, but they just don't want it linked to performance. Okay, which you can certainly understand that position, why a teacher might not want a linkage with performance. Okay, so I'm gonna be in this world in Chicago Heights. The reason why we didn't do these experiments in Chicago Public Schools is because a union would not allow it. We tried dozens and dozens of times, I was even close friends with Ron Huberman, who was a CEO of Chicago Public Schools, and we could not convince CPS teachers to do an incentive paid program. Even though I promised to bring in $10 million to the city that was free. They did not receive that $10 million because they were not part of our program. So this was not taking existing money, this was bringing new money from a donor into the pool. Okay, and they would not agree to do it, so we went to Chicago Heights. Now here's what we did. We told them we're looking for improvement on standardized test scores. See these are high stakes standardized tests called the ISAT that kids will take. We told them you could earn as much as $8,000 over the year. On average, each teacher will earn about 8% of their salary or $4,000. Okay, so that's the particulars right away. We're talking about high stakes. We're not talking about a survey that's non-incentivized. We're talking about going into a school district and looking at real stakes. Now, what we're gonna be innovative on is how we frame the incentives. Here's where we will add some behavioral economics. So we will start with an incentive scheme that we all know about. Typically, what do you do? You work for the year. At the end of the year, you're paid a bonus. Every firm doesn't like that. We do at the University of Chicago and the Department of Economics. As a department chairman, I have a small account that I can give out bonuses at the end of the year. So you work, work, work. You're evaluated, you're given a bonus. That's how we're gonna set up what I call the gain treatment in this experiment. Now, the novelty is what I call the loss treatment. So here, what I'm gonna do is with a new group of teachers, I'm gonna give them $4,000 in September and I'm gonna set up the rules exactly as it is in the gain treatment. We're looking at standardized test score improvements. We can debate whether standardized tests are the right metric, but that's not the point here. The point of this exercise is try to look at whether incentive schemes can work to induce teachers to work harder. So what I'm gonna do is I'm gonna give each teacher $4,000 and I will tell them if your kids do not improve, you will have to give us back money at the end of the year. Now, when you do something like this, your lawyers have to talk to their lawyers and we had all that worked out. If your kids did not perform, you had to write us a check back. I call this the loss aversion treatment. Psychologists for decades have talked about losses being felt more than comparable gains. So what I mean by that is if you lose a dollar, it hurts a lot more than the gain that you feel if you gain a dollar. So if you view this a bit in terms of introspection, a lot of us, I think, have this feeling initially. It really hurts to give something up that we own. So we're trying to leverage this aspect of behavioral economics to see if we can set up an incentive scheme that works, okay? So this is a very easy experiment, a very easy field experiment. We have three groups, group one, traditional bonus, group two, loss aversion, and group three, these are the unfortunate teachers who receive no bonus at all, okay? Let's see what happens. So what I have here is I have the racial achievement gap right there. That's the difference in standardized test scores between blacks and whites. This is what's called the racial achievement gap, what we're all trying to understand, and we're all trying to raise everyone's boat and get rid of that racial achievement gap. On the Y axis, this is in standard deviations, okay? So what this tells us is that whites outperform blacks by about a third of a standard deviation on these standardized tests. That's known as a large effect, okay? So just to put my effects in perspective. Now what I have down here is the gain treatment. This is how students who were in classrooms of teachers who had the gain bonus, how much better do they do than the control group or the students who were in classrooms of teachers who had no bonus? As you can see, it's only about 0.06 standard deviations. That's a very small effect. Okay, so all of the press clippings that we went through, they're right. When you do a standardized test experiment on performance pay, the regular way in which we think about incentives, that way doesn't work. But guess what? If I give teachers the money upfront and threaten to take it away if their kids do not perform, those kids are value added out of the roof. And what I'm saying here is about over 0.2 standard deviations. That's close to one year of schooling in this community. It's two thirds of the racial achievement gap. If your kid just was lucky enough to be placed in a classroom of a teacher who received the loss of version incentives, they went up a very large amount in relation to the control group. Two thirds of the racial achievement gap is taken out here. Incentives do work in this case. We just need to figure out what is the best way to use them. And then they'll work in spades. Now, one message is very clear here. It is very difficult when you enter a child's life, when they're an eighth grader, ninth grader, or a tenth grader, it's very difficult to change that person's life in a deep, deep way. What am I talking about? Hard to make a ninth grader who's reading at a third grade level to be a STEM major. Very difficult. What you can do is you can fundamentally change their lives by getting them to graduate from high school. And that's important. And get them in a job as a mechanic or a truck driver or a secretary. And I say that with the greatest respect because my family, those are the occupations of my own family. Okay. Now, what's also key is if you look at today's technology, I suspect that it's very difficult to think of a time that we have wasted more human potential than we are right now because of our lack of understanding public education. This is a conjecture that's very difficult to test, but I will put it out there. I think you need to go back to the dark ages to find a time when we wasted this much human capital potential. So what did we do about it? We started our own pre-K programs. We went back to Chicago Heights and we wanted to learn about what are the best ways to teach three, four, and five-year-olds because we fundamentally want to change the course of their lives. And that's where we began to invest resources about six years ago in early childhood education. Let me tell you a little bit about one of those experiments. Okay. We started off by saying, let's create two groups. And these groups will be kids who will go to an all-day pre-K program in two separate buildings that we run and we started. What we didn't start are the curricula that we use. In group one, if you go to this school, you are in what's called tools of the mind. Has anyone ever heard of tools of them, the curriculum tools of the mind, good? This is supposed to push executive function skills. Fogatsky talked about this in the 30s that this is a way to push what some scientists call non-cognitive skills. So we have a school just set up for three, four, and five-year-olds to teach them non-cognitive or executive function skills, self-control, patience, teamwork, et cetera. If you're in group two, you're in a different building and what you go to is an all-day pre-K program under the curriculum called Literacy Express. Anybody ever hear of Literacy Express? Okay, several people. This is advertised as the meat and potatoes curriculum for cognitive skills. Okay. Those are groups one and two. Groups three and four. We're gonna design our own curriculum for what I call a parent academy. Here, what we're gonna do is we will not teach any of these kids directly. So if you're randomized into group three or four, we never teach your child directly. What we do, however, is once every two weeks, we meet with you in a small group and we teach you about how you should be teaching your child and approaches to doing it. The only thing we change across groups three and four is that in group three, they are incentivized to come to the program with cash payments. And believe me, you need that. We tried it in pilots without cash payments. Nobody shows up. Okay. In group four, a bulk of the incentive is put into a college tuition fund. So if your child decides to go to college, you will then draw down that fund and use it for tuition relief. If your child does not go to college, that fund comes back to the foundation. The idea between three and four is we're trying to look at what are the best ways to stimulate not only short-run investment in child, but also long-run investment in child. So group four, long after our program ends, remember, these kids are coming at three, four and five years old to us and then they're going out to public schools. So we think that group four will cause parents to keep investing. Why? Because college is now cheaper and when their kids go to school, they will get a better deal. That's how economists think about using prices. 728 families, unfortunately, are in the control group and they are not receiving any formal pre-K or parent academy from us. Some of them will go out and get their own help and they absolutely should. We measure that as well, but we do not give them any formal training or programs in our program. Okay. So let's take a look at some of those results. What I have here on the y-axis is percentile change. Let me explain what that means. The kids in Chicago Heights are roughly at the 33rd percentile when they come to our program. You can say, well, what does the 33rd percentile mean, John? It means if you line up 100 kids, random kids around America and give them a cognitive test, our kids would be 33rd from the bottom. Okay. So what I have here is I have each of the four programs and this is against the control group. And the big bars occur in the first four months of the program. So as you can see, preschool literacy express, within four months of the program, we take kids from the 33rd percentile all the way past the 50th percentile. So in four months using cognitive tests, they go from a south side resident to a resident of the north side in terms of their cognitive test scores. And what's interesting is that the literacy express kids outperform the tools of the mind kids exactly as the developers of the two curricula tell us they should. The parent academy, you have results, as you can see here and here, but they're a little bit smaller than preschool literacy express, a little bit larger than tools of the mind. But again, what's interesting here is the lion's share of gains occur in the first four months of the program. Now when you look at non-cog tests, here what's interesting is the preschool, tools of the mind kids. Just as advertised, they go up considerably in executive function skills. The other groups do well too. You can see parent academy, cash and college and literacy, but the preschool tools as advertised actually increases more within this experimental period. So now the takeaways, after years one and two, what we took away is that, wow, we can get a bulk of these gains in the first four months of the program. And that's exciting from a public policy viewpoint. And I'll get to that in a moment. Now what's interesting is that the literacy express kids just go off the chart in academic growth 20 months in one year, just by being part of our program. They catch up and pass the average three, four and five year old across America. Tools of the mind kids, they're really increasing their non-cog or executive function skills. So effectively what I have here, all these kids are now in fourth, fifth and sixth grade as we speak, what I've fundamentally done is I've randomly put some kids as executive function kids and some kids as cognitive skill kids. We don't understand what skills we should be giving our kids when they're young, this research will help to answer that. And I'll get back to that in a moment. Now when you look at the parent academy, what's interesting about the parent academy is that our parent academy results just explode for Hispanic families. So the entire result from the parent academy comes from Hispanic families. If you look at African-American families, it's close to a zero effect. And I say those two groups because that represents about 90 to 95% of our sample. Okay, so fundamentally our parent academy is helping Hispanics while it's not helping our African-American students. Now that's important because as policy makers, many times we lump those two groups together and say we need to do this for minorities. Time after time, what we've been finding is that there are dramatic differences between programs we need to use across Hispanic and African-American kids. In fact, Caucasian kids are kind of in between. And that's important to know, we would not know that had we not done this experimentally. So now after those first two years, I was sitting on a lot of human capital. I had a bunch of teachers who had taught tools of the mind. I had a bunch of teachers who had taught literacy express. I had developed a parent academy. So I had people who developed the curriculum in this great parent academy. So I had all of this human capital around, what do you do? You wanna make use of all that and combine it all. So in years three and four, we used a combined curriculum that took the best parts out of tools of the mind. It took the best parts out of literacy express. It took the best parts out of the parent academy. And we knew the best parts, why? Because we tested for them through the two years. We asked the teachers, what do you think works? And I promise you, that's the worst predictor about what actually worked in the data. It's not a good predictor. We asked them what works. We found out what worked. They tried new things as we went along. And now we've developed this curriculum based on field experiments, which we call COGX, which means cognitive executive function, skills that we've run for the last two years. Now you can ask, well John, you do field experiments. Now that you have your curriculum, what are you going to change experimentally to figure out things about what works and why? Well, remember the timing. A key result that jumped out of our data is that we get the majority of our effect in the first four months. And wouldn't it be interesting if we could set up programs that would run in the summer before kids started kindergarten? Why? Because buildings are empty. Teachers are unemployed. As an economist, I don't like idle resources. We need to utilize resources if it turns out that we can get big gains in the summer months before kindergarten starts. And note that that's also nice because you get rid of summer depreciation, which is a key moment in a racial achievement gap. So can we leverage those resources and science to think about a three month program in the summer before kindergarten starts? And if that's as effective as a nine or a 12 month program, we have just developed a really nice policy initiative. Much cheaper because we have a lot of idle resources. So that's exactly what we did. In years three and four, we had about 1500 children, which is essentially all of Chicago Heights. So we have nearly 100% by this time, it's a free program, everyone's excited. So we're doing our curriculum COG-X across groups one, two, and three. The only thing is, is that it varies. Group one is for 12 months, group two is for nine months, group three is what we call kinder prep. And that's the parent academy in an all day pre-K just over the three months of summer. And then group four is our control. Let's see what happens there. What we find is that COG-X, the lift that we observe there is much better than years one and two, which makes sense because we're combining parent academy and all day pre-K. Interestingly, our early evidence so far suggests that kinder prep, the three month program, does as well as the nine and the 12 month programs. Now to me that's fascinating because now we have a legitimate public policy proposal that is very cost effective. Parental investment here is slightly above the parent academy from years one and two. What that tells us is that when you treat the child and the parent together, the parent gets more invested and they invest more in child. So rather than being substitutes in the production function, they end up being compliments, which is important. Now the takeaways here is I have used, and now let me be clear. When I say I, I mean we, and when I say we, I mean they. Because I have plenty of research partners, Steve Levitt and Roland Fryer and others. Okay so let's just put that caveat. What we found is through this field experiment is that we can provide significant expansion of COG and non-COG skills. After four months in our program, inner city kids pass the average American in both COG and non-COG. This leads to the statement if everyone is given a fair opportunity to excel. I believe this gives us the best chance to grow the world economy. So where are we going with this? We're not gonna stop here. We have resources in place to follow these kids over the entire life cycle. These programs might affect crime rates, they might affect teen pregnancy, they might affect the education. They could discipline by the time the kids reach the third grade on some dimensions. We need to fundamentally understand that and what we're doing right now is we work with the public schools to see why reversion occurs. So if we treat three, four, and five-year-olds, but then they're the same as everyone else by the third grade, we need to learn fundamentally why did that reversion occur, we're doing that right now. Because that's what's happened in older programs called Perry Preschool and Abbasidarian. And people don't really understand why that happened. We're learning that as we speak. Now what we've also done is we've used CHECK as a small-scale society to run a lot of experiments, the Marshmallow experiments that were famously done by Michelle to learn about patients and how our programs affect patients, these sorts of studies. Now I'm gonna switch gears a little bit and say we fundamentally only touched the tip of the iceberg here. Let's look at a few other questions like, why do women earn less than men? When you look at mounds and mounds of data, what you find is that women earn 10 to 20% less than men in labor markets. And then when you control for a bunch of factors like time out of the labor force, occupation, educational status, you still have a significant gap of about five to 10%. What we've been looking at is the fact that men and women prefer different kinds of incentive schemes. This started as lab experiments from lab researchers, Uri Ganesi, Miriel Nederle, Lisa Vesterlund, Aldo Rusticchini from UM. They found in the lab that women shy away from competitive settings. We find the same sort of thing in the field in that women do not like merit pay incentive schemes as much as men like them. And if you don't like those kinds of schemes, you're much less likely to select into jobs that use those types of schemes as incentive pay. So what do women do? They opt out of those types of jobs. The problem is those types of jobs tend to be the highest paying jobs in our economy. So then you have a fundamental difference in pay between men and women. Now we ended up doing a large scale experiment on bargaining to try to figure out how do people bargain their first wages? When you start a job, if you take nothing from this talk, remember that basically every $10 or $20 you get in your very first negotiation ends up rising exponentially because raises are built on your base pay. So we wanted to explore why do men bargain the way they do and why do women bargain the way they do on their first wage contract in a job? So what we did is we went to Craig's list and we advertised two identical jobs. These were office support jobs and they were real jobs. We hired real people to the University of Chicago. But we put these ads out and thousands and thousands of people applied to our ads. We did it in two ways. The first way what we said is wages are negotiable. In the second job, which was otherwise identical, we left that sentence out of the ad. Guess what happens? When it's not there, men bargain like mad and women shy away from asking for more money. When that statement is in the ad, women bargain harder than men. So if they're told it's okay to bargain, women will bargain like mad. If you set it up to be ambiguous, just leave the sentence out. Everything is otherwise identical. Men bargain much more than women and guess what, there's another twist to it. The men who are of the lowest quality bargain the most in that setting. Yes, the lemons are bargaining the most when it's ambiguous. Now, you can say okay, John, that's a really interesting fact. Why is that the case? So we've also been going after questions like nature versus nurture. Why do women have preferences like that and why do men have different types of preferences? So we've gone to a lot of societies around the world. One of my favorite is a matrilineal society called the Cassies. Here, this is a fundamentally different type of world than the Western world. As we were driving out to the villages in India to see the Cassies, there was a billboard that kept coming up along the road. Same billboard. We asked our taxi cab driver, what does that billboard mean? What does it say? He says, ah, it's the men. They're arguing for equal rights again. Wow, that's interesting. That's an interesting society. You knock on the door of a Cassie household and the man answers, he bows, he takes you back to the woman of the house. So we wanted to learn about how do people behave in that kind of setting? How do men behave in bargaining situations? How do women behave in bargaining situations? The men there, they will tell you, we are sick of being breeding bulls in babysitters. You know what happens? They're women act like our men. They're men act like our women in these games. I think we're born, nature's important, don't get me wrong. We're born with a band that says, how risky or how competitive do I wanna be? And society tells us where we should be in that band. And then we go there because we do not wanna suffer the consequences of not being right socially. This sort of came up this morning in Paul's talk about what society tells us we should be doing. That's important. No doubt it's important. Why do people give to charity? We've been doing this kind of research now since 1996, looking at what causes people to first of all, give money to charitable causes and then remain committed to charitable causes. What's interesting is when you look back at all the old writings, it's always about altruism. It's always about someone doing a favor for someone else, someone helping a poor person, someone helping an unlucky person. In our data, that factor is not key. What is key is that people tend to give for purely selfish purposes. They tend to give because it makes them feel good. It's called warm glow in our literature. Now I know I'm a University of Chicago economist that comes in talking about selfishness. Don't get me wrong, altruism is alive and well. I'm just telling you that a key contributor to charitable giving is selfishness. And that's not a bad thing. It's a bad thing that we didn't know that. It's a good thing in that now we can leverage that understanding and make the world a better place by getting great charitable organizations to raise more money and overcome these problems that we face as governments devolve or decentralize authority of providing great services to the poor, great services on the natural resource side. We need to have someone step up. And that's 501C3s, charitable organizations. They need to fundamentally understand the science of charitable giving. We've been going after that question as well. What about discrimination? This is ubiquitous. In economics, we have two major models about why people discriminate. One model is due to a famous Chicago economist named Gary Becker who wrote in his dissertation in 1957 about bigotry and animus in that people discriminate not to make more money. In fact, he said some entrepreneurs will even make less money to cater their own prejudice. They will give up money to hire the man or give up money to hire the white person even though they would make more money hiring the other person. It's what Becker called animus or bigotry or a taste for discrimination. That's one very nefarious type of discrimination. Another one is what Pagoo talked about. He called it third degree price discrimination. Today we call it statistical discrimination in the economics literature. What does that mean? In an effort to make more money, people discriminate. So they might bargain harder against women who are buying a used car. Why? Because it's well known amongst used car dealers that women do not like to bargain. So then what do they do? They start at a higher price and they bargain harder. And in the end women pay more than men for used cars. That's why CarMax came in and a big part of their market early on were women. People who wanted to avoid the hassle. Those are two very different types of discrimination and it's hard to figure out what kind of discrimination is happening in markets unless we have field experiments. Nearly impossible. Looking at mounds and mounds of data. Economists have tried it for decades really, really hard. We've been able to disentangle it using a series of field experiments. Now what we've been finding is that animus is becoming less important over time. In third degree price discrimination is becoming more important through time. Now this is not a great sign because guess what? Is big data come around. Mounds and mounds of data come to firms. What do you think those firms are doing? They're looking at, for example, if you go to Amazon.com they know exactly how many other places you visited before you arrived at Amazon.com. And then they can give you prices based on that. They're called customized prices. And I suspect those customized prices are not to give you as much surplus as you can get. Why do people vote? We talked about that earlier as well. Our hypothesis is that many people vote to tell others that they voted. You can say, well, wait a second, John. I can always just tell people on the Wednesday, Thursday after the election, I can just tell them that I voted. You can. But what we estimate in this particular paper is you then receive what's called a disutility or a hit to your happiness. People are averse to lying and they will pay a price to avoid lying and the price is they will go and vote because they anticipate people will ask them and they're gonna be unhappy if they lie to them. Many people vote because they realize somebody's gonna ask and they wanna tell them the truth. Now how can we use that? The Obama campaign used that during the last election to do their get out to vote things on people who they thought were gonna vote Democrat. It's exactly, this is one of the tricks from our paper that they used. One great big externality that we oftentimes talk about is climate change. You might say, well, John, you've talked a lot about how field experiments can save the world, but what about a great big externality like climate change? How are you gonna take on that challenge? Well, what we're gonna do is we're gonna go to households and we're gonna say households are a major contributor when you look at the data to the climate change problem. And we will go to households and try to figure out why aren't households adopting green technologies? Why are they leaving dollars on the table and what can we do to induce households to adopt green technologies? So what we did essentially is we started out looking at social norms, telling people, seven out of 10 people in your neighborhood have green technologies, why don't you? That works for the very first adoption, but it's really difficult to use that trick for deeper adoptions. What works well for deeper adoptions are price cuts. So now this tells you as a policymaker to get somebody involved in the market to start, we can use things like social norms, injunctive norms, basic norms, but if we want deeper adoptions, we should be using prices. And in that way, non-price and price incentives serve as important complements to make the world a better place in the area of the environment. Now let me end here by telling you a bit about, in my life, our currency as academics is to write academic papers. In fact, Orly handled one of my very early field experiments and it was kind enough to put it in his journal, which is the best journal in the world called the American Economic Review. So Orly is a major part in the late 90s served to expand field experiments in a very, very important way because he put our field experimental work in the top academic journal is the editor. Before Orly got a chance, I submitted my first paper in 1994. And here's what my referee report said. This author clearly does not understand experimental economics, which is best done in the lab. I strongly advise rejecting this, quote, field experimental paper. They said that to a grad student in 1994, everyone else is telling me what you're doing is dumb. Now I have a referee report telling me what you're doing is dumb. Let's fast forward to a very recent referee report, same journal. Here's what they said. This author performs a field experiment which clearly is a preferred approach to answer questions within economics. So the battle, the big battles won, but I still advise rejecting the paper. The particular war was not won, but the battle certainly has been won and we've moved. And I wanna say thank you to the organizers. This has been tremendous. Audience, thank you very much for your attention and I'll be glad to take questions.