 Well, you know, I think that was just a great, great representation in it. I have this deep suspicion about field experiments, because in all those cases you had this control group, and the control group was being screwed. So there's an ethical problem with field experiments, but I'm sure you're very, very sensitive to that. But as an economic historian, I want to be on your side, and here's how. I want you to agree, as the chair of the economics department at Chicago, that Chicago should have more economic historians, because when I was at Chicago, on the faculty there, we had three out of 23 on the faculty. And here's the reason. It's the same reason that you gave, and I want you to react to it. We're all looking for more facts. You're looking for more facts by learning in detail about these countries' earliest looking for more facts by going to still another McDonald's. We're all looking for more facts. And it's a bias in economics against historical facts. Come on. All our facts, in a sense, are historical, because everything we know is now the past. But I recommend that you go back to the Middle Ages, as I have. Go back to the 18th century, and furthermore, not just in one country, but in China, in Japan, in Africa, in South America, and see the whole. Let me just give one example of what a powerful tool of inquiry economic history can be, schooling. Why did the United States in the 19th century have the best K through 12, or certainly K through 8 schools in the world, except maybe Prussia? Well, it was because women, middle class, moderately educated girls, really, had no other occupation they could do. They were, say, they were farm wives, or they weren't, didn't much matter. And the occupation they could get into was elementary school teaching. So you had, I don't want to insult the teachers here, but you had these really bright girls going into education. Concrete example are nuns, Kulu Catholic nuns. The Catholic elementary school system, I'm personally acquainted with a couple of these people, was staffed by really intelligent, hardworking young women who had found a vocation. So that's what you can learn from history, and read my books, John, and you'll get a whole bunch of books. Hold them back. Can I, can I have to make a comment? Read my books, read my books, but that, that's another dramatic source of insight into economics. I have to make a comment here about John. I think it'll be interesting for the audience. It's not substantive, unlike the comment that Derger made. John made some nice comments about me, that I was open-minded, I took it to mean. But I should tell you, I think the audience will enjoy this. The first time I met John, John is an unusual economist, not just at East from Wisconsin, that his PhD is from University of Wyoming, and he started as an environmental economist doing experiments that were, they were self-financed, field experiments, self-financed, using his baseball, it was a baseball card collection, he had a message, baseball, he did some experiments in which he made a lot of money because he found anomalies, people making mistakes where he could self-finance his experiments. Anyway, the first time I met him, he invited me to speak at the first place, I think it was the first place you were teaching, Central Florida State University. For those that don't know it, that's near Orlando, so think Disney World. I go down to John's place, he scoured up a great audience for me, I gave a talk, and then we went to his office. Now this is an amazing story, you just aren't going to believe this, but it's true. We went into his office, this tiny little cubicle, and in this office it was packed with trophies, just trophy after trophy. I looked at John and I said, what are all these trophies? He says, these are the trophies from the waters, from our championship water skiing team. I said, what are they doing in your office, John? Oh, I'm the coach of the water skiing team. So I said, John, I'm amazed, I didn't know you could water ski. Oh, I don't know how to water ski. I'm not the coach of the water skiing team because I know how to water ski. I'm the coach of the water skiing team because it's not an NCAA sport. Now for those in high school who don't recall, if you're a member of the National Colleges Athletic Association, you cannot pay your athletes. So therefore, John explained to me, no faculty member at Central Florida State University would coach the water skiing team because it's not an NCAA sport, and you can pay your athletes, which was probably the origin of many of the tournaments. Instead of, so thank you very, that's very kind, and that story is correct. That was the coach of the UCF waterski team. I want to come back to, so the University of Chicago, some people think about history as, I'll learn about it when it repeats itself, and that's not how the University of Chicago, I think, thinks about it. I think if you have a person who is an excellent scholar, we'll greet her with open arms. We don't have a problem with that. Now fundamentally, you're right that economic history has evolved to a place where it's not as theoretical and it's not as, quote, empirical, is what other areas in economics. It depends on what you mean by empirical. If you mean it's not stupid econometrics to say, on completely irrelevant tests of statistical significance, then I guess you're right. But not to get back to the substance a bit. So I certainly understand that we have to think about ethics when we have experiments that you're running when people don't even know they're taking part in an experiment. I've written about this in the magazine called Science a few times about, all these have IRB approval, which means the college approves them, but you still have to have a higher level than that. And what the way I think about withholding treatment is that I'm sure glad that we've had medical trials because if we would not have started medical trials in 1964, my dad would not know what medication he should be taking for his heart problem. Now, we fundamentally have made a decision that in the medicines we will give a sugar pill. That's really bad, right, giving a sugar pill when you think you're getting the cholesterol pill. But we do that why it's not because we don't understand intragenerational fairness. We all understand we want everybody today to be better off and we want fairness today. But that doesn't mean that you ignore intergenerational fairness, which means we need to do the best we can to learn and use science about what works so our kids and grandkids and their kids can have a better world. We've done that in medicine. We've made that step in medicine. I don't understand why we would not make it in education and make it in the other parts of the science. It has been a dramatic success in medicine, no doubt. We need to use that same thinking. I'm not saying be unfair or be inequitable, but by the scientific design to have a control group, we need to withhold treatment from some people. Now what does that mean? I'm not saying take tax dollars from them. I'm saying all of my experiments are funded either by my baseball card collection, as Orly said, or by a philanthroper who gives money where that school district would not have those dollars if it was not for the philanthropist. I'm not going into a school district or anywhere else. And withholding resources that they would otherwise get. My field experiments, I always have a mantra that I want everybody to be better off in my field experiments. I understand that. In expectations. And that's where I think I stand by my word. But I must say that you can't take medical research as the gold standard. That's what they talk about, the gold standard. Here's an experiment that I can wholly approve of. Walter Reed was the doctor who discovered that yellow fever, which was making it impossible to build the Panama Canal, was transmitted by mosquitoes. Here's the experiment that this guy performed. He got a bunch of mosquitoes from Panama, put them in a box, and he stuck his own arm into it. And he got yellow fever. Now, there is an experiment, I'm not against all field experiments, but I've heard of shocking field experiments where, for example, in China, because they couldn't get it approved in the United States, they would give some school kids who couldn't see very well glasses and others not glasses. And then I ask if the kids that had glasses learned Chinese characters better. Now, if you can't see, you can't learn Chinese characters. And so there's something deeply unethical about that. From what you've said, it doesn't sound to me, and I've read some of your work, that you have this problem. But I don't think you should hold up medicine. There was an old New Yorker cartoon a long time ago that had a tombstone. John Smith, member of placebo group. So I'm curious, and so the powerful research was early childhood education. And there's a lot of experiments around the country in New York City and Chicago elsewhere. And the data is compelling. I mean, oftentimes, I don't think you do the exact same experiments as what you're doing. But it really does show results. But what seems to be the problem, I just wonder if you have any insight or thoughts about spreading it. Yeah, absolutely. Like some of the programs, why aren't they in Minneapolis? Why aren't they in St. Paul? The ones that in Chicago or in New York? Absolutely. So I think it's an important question. When I answered the policy world, I had two things in mind. One, I wanted to test economic theory and deepen our understanding of economic theory. And two, I wanted to create programs that were scalable, that we could change the world, like COGX, as I mentioned. Now, the scaling problem, I call it the science of using science. So I'm writing a paper with that title. It's a very interesting problem. And you can start off by asking, John, you got an effect of 0.2 standard deviations from your treatment. Do you think if you scaled that up to Illinois or the US, would you expect to find that same effect? And we're writing, first of all, a theoretical paper on scaling and what it means. I think, first of all, there's an inference problem there. Because we have to understand that when we reject nulls, that takes account of p-values, but you also need to take account of your priors in power. And if there's a mechanical problem that if you don't understand the power of your test in your priors, you can't just look at p-values or even significance levels. But after you put them in that model, there's a mechanical reason why it won't scale. Because it's either a false positive or you got a good beta draw. That's why I presented it here. If I'd have gotten a bad result, there would be no presentation on that part of the study. We know that. Okay, so you have an inference problem that you, first of all, have to overcome. Secondly, you have properties of the population problem and a properties of the situation problem. With population, it could start with maybe I got really good teachers. And if you wanted to scale it up, the economics of the problem says there aren't as many good teachers to go around. Deirdre stole my line again. What I oftentimes say in front of an education crowd is I say, you know who's ruined education? Women. And then everyone goes, oh! But I say, because they're no longer discriminated against in markets and they can get the returns that they deserve. And it's also minorities, by the way. It's not just women. And it's the fact that the very best who are becoming school teachers, my mother-in-law is one of them, in the 50s, 60s, and 70s, no longer do that. They go to Wall Street and they're head fund traders. Or they're Cheryl Sandberg at Facebook. She would have been the most glorious teacher in the 50s. Now she's a CEO of Facebook. And I like our new economy like that. So you have an economics problem on the population of, let's say, the fidelity of the instrument. Who's gonna actually run it? And then the population of people. You have to be careful because you have people selecting into an experiment. And it's those people who you randomize over. So that gives you an interesting parameter that everyone who agreed to be part of your experiment, you find treatment and control. But what about the people who did not decide to be part of the experiment? Usually we use economic theory to say, if it's like this here, it's gonna be like that there. I use natural field experiments where I go to the real world. Thanks. There's no selection problem because I randomize over people who do not know that they're taking part in an experiment. So I don't have this problem of people selecting into the experiment. That's what natural field experiments help you overcome. Economic history. I wanna learn about today, not 200 years ago. No, I think that 200 years ago can help us today, but I wanna learn about today as well. I want both. I think you're just trying to point out that we want economic history. I actually want both. I think we can learn from the past, but we can also learn from what's going on now. And what I want is your experiment throughout the country. Absolutely. So right now we are setting up two programs. One called districts of innovation where we're gonna have a call where there are gonna be districts around the US who we coordinate the experiments that they run. So now we should replicate when we should replicate. We should innovate when we should innovate. We're also doing a COG-X nationwide experiment as well. I love it. So those two things are going on right now. I don't have any data on either. I love it. I've got a question. John, can I ask a couple of things? Daniela, you win. Unless you've got Daniela, go ahead. I'm gonna ask a different question about a different problem that our audience has raised. Could you please speak to recent scandals of teachers choosing to raise student test scores? Yeah, absolutely. So I think anytime you set up an incentive scheme, you always have to worry about unethical behavior, especially if the incentives are strong. I think what that suggests to me is that the incentives are strong for behaviors, but you have to make sure in our particular experiments, we have outsiders come in and do the standardized test. So our particular experiments are immune to that, but certainly other people, Steve Levitt and Brian Jacob have a nice paper that shows how Chicago public school teachers cheat to increase their students' test scores because they knew they were being evaluated based on that. That will certainly happen. And when you put in high-powered incentives, it will certainly be the case that people will do everything they can to get those incentives. We just have to make sure experimentally that we're finding a treatment effect rather than a cheating effect. Okay. This is when one of our high school students is the bargaining for wages. The only cause for the wage gap or are there definite pay differences purely because of it being either a man or a woman is applying for said jobs? Yeah, that's a good question. I think when you look at the wage gap, if you put up a figure of the wage gap across OECD countries, you'd find countries like Korea would be a 40% weight, women earn 40% less than men. You go down to the OECD national average, you have about 17.5%. That's about where the US is in the raw data. And then when you start controlling for things like the occupation, the gap lessons, time out of the workforce, the gap lessons, the characteristics of their education and their human capital, the gap lessons, and you get down to about 10 cents every dollar. In there, you have this component of, I believe, discrimination as part of it. Not only discrimination in the initial job, but also discrimination and promotions in the way that women are treated in many workplaces. I also think the fundamental way that we design are incentive schemes, which are predominantly designed by men. Men have designed job ads, have designed the way that people get paid and there's no doubt that they've designed them in part so they're better off. Now, I think all of that leads to this notion of it's a bundle of things. I think the behavioral elements that I raised are a part of it, but they're by no means a large or a majority part of it. I think it's a significant part, but a part of it. Okay, let me just add one thing to this that hasn't been mentioned. Really, can you put your microphone a little closer to your mouth? Isn't there wrong, can you hear me? Can you hear me now? Or should I hold on to it? Let me just add one thing. People mentioned the gender gap. The gender gap has narrowed considerably. One of the reasons why there has been an increase in inequality amongst individuals, but it's much less increased inequality amongst family incomes. And the reason for that is because while there's been an increase in inequality within the male and female group, the gap between men and women has actually narrowed. So for the high school student here, this is not nearly as big a problem as your mother faced. It's not that there isn't a problem, but it's not as bad a problem as your mother faced. That's correct. Related to that, another question from a high school student says, aside from the wage gap, do you feel that men have the tendency to obtain leadership positions because of discrimination or other causes? I think that's certainly part of it. I think when you look at the leadership positions, it's people on the panel probably know the stats better than I do, but of every 10 CEOs, it's certainly nine and a half, 9.9 are going to be men. And you can say, why is that the case? I think what Orly alludes to is that's changing, but it's changing too slowly. I think that's certainly clear. I think there's still an old boys network. There's no doubt about that. There's an old boys network and academia too that we try to overcome. But I think it's a confluence of events. Life is incredibly unfair for academics who want to have kids, but then they're evaluated in their first five to seven years about where they'll be getting tenure and the maternal and paternal instincts, I think are fundamentally different. And you have that in the labor force too more broadly because early on in your job, you establish where you are in the pecking order. I go out to Uber a lot right now and I see their corporate setting and as new people come in hundreds of people come in every day, the doors is new employees of Uber. What you have is there's a pecking order right away you can establish in the first several years, first several weeks, months, years about where you are in the corporate system. And I think that's fundamentally needs to be transformed and thought about to in the end narrow this gap to what it actually should be, which is zero or pay people their marginal product. Women right now are much more productive for what they've been paid. I'm gonna come in with a last question if I can Larry off Twitter and that is a question about whether your experiment in Chicago Heights attempted in any ways to deal with the segregation of communities in that neighborhood. That's a good question. Right now what we're doing is we're looking at the social networks in the community, what they were like before the experiment and whether our, for example, our parent academy treatments led to different networks because we put parents who would otherwise not be in the same room, whether they ended up being in the same room randomly and then did they keep those networks? As I mentioned, our kids are fifth and sixth grade the first years. Did our experiment fundamentally change the social networks which changes the segregation? I think this leads to some foundational questions about composition of cities and what we can do as policymakers that isn't just moving to opportunity like we've tried that's so dramatic, but if there are smaller things like parent academies in settings where people will go and these networks will form endogenously in ways that can be very beneficial to society. So the question's a great one. I don't know, but we are looking at the networks as we speak. Is there time to follow up on that or? Certainly. It's a network question in the end and it's back to African schools and the problem that African teachers tend not to show up for class. Now, the experiments at the moment that are being thrown at this are bonus pay if you show up. I'm radically uncomfortable with that because I believe that norms matter and if we introduce bonuses for what should be a fundamental part of the job we're undermining a norm that has to be established. So I'm uncomfortable with that, but if what we've got to do is sort of within and the other striking thing in Africa is that some schools work wonderfully and some are terrible. And so it seems to be the culture amongst the teachers that differs. You've got good cultures and bad cultures. Now, changing the culture of teachers in a bad school is the whole network that's wrong. And so we need, I think, not experiments on bonus pay, but experiments. How do we change the network? And I need some ideas. So let me comment on that. So first of all, let's start with the last point. Experiments that I did not talk about here, we actually put teachers in teams and then we incentivize the team production in a way that we're trying to get out there. Teamwork is important. You should teach each other what works in the classroom and that's a way to, let's say, transfer technology or transfer know-how. And I think you can do that through markets or incentive schemes and we're trying to induce that as well. Now, about intrinsic motivation, I think anytime people hear about, you're adding dollars to something that people should be doing anyway. They find that repugnant and they say, look John, why are we adding dollars here? They should be doing this anyway. And then the more refined argument of that says, John, your external incentives or your money will crowd out intrinsic motives that they have. That's right. So let's start. We have a problem because these students don't have intrinsic motivation. If they had intrinsic motivation, it would be fine. We could think about getting the intrinsic motives to come out and flourish. That was part of our tools of the mind. We wanna give kids early on elements of intrinsic motives that their parents are not giving them. So there's really nothing to crowd out when you talk about using money might crowd out. Although that's an interesting scientific hypothesis. I think in settings where you have intrinsic motives that can happen. And I even have some of my own data where that happens. But I think much the same way we think about we want people to stop smoking. We want them to eat in a more healthy way. We want them to take their meds. What we do is we use taxes and subsidies. We want people to engage in buying cleaner cars or using appliances that are less energy intensive. We use prices. I'm not saying that prices are the only tool. I've told you I'm a behavioral economist. I believe in non-pecuniary incentives. But there's still a role for prices when the intrinsic motives aren't there. Prices are tried and true. Demand curves are downward sloping. Supply curves are upward sloping. And if we use them in other walks of life, I think it behooves us to at least try it. And when we try it, we've learned about does it work? And then as a society, we decide is that something we want to incentivize using pecuniary incentives, using non-pecuniary incentives, using parental programs? That's what we decided as a society. I don't think we can make those decisions until we find out what works and why. And then we can make those normative decisions. So I think of myself as a positive economist who I'm trying to generate data to generate facts. So then people who are better than me at making normative claims can take my facts and hopefully generate better normative arguments. Thank you. Again, we are slaves to the clock, but we want to be able to reconvene at three o'clock to hear Deudre McCloskey. So thank you very much for our panel and our audience. We'll see you back here at three o'clock.