 Today, I am speaking to Phillip Tetlock, who quite simply is one of the greatest social scientists in the world. Up until now, we've been doing conversations with Tyler face to face. But for obvious reasons, Phillip is in Philadelphia. He teaches at University of Pennsylvania, and I'm here in Arlington, Virginia. Let's just jump right into it. First question, Phillip. With our forecasters, do we want accuracy, or do we want them to be a kind of portfolio to make us more aware of extreme events and possibilities? I think we want a lot of things from our forecasters. And accuracy is often not the first thing. I think that we want, we look to forecasters for ideological reassurance. We look to forecasters for entertainment. And we look to forecasters for minimizing regret functions of various sorts. So that we would really regret not having anticipated X, Y, or Z. So we want to pump up the probabilities of those things. But if we take, say, the coronavirus, if we had had a few more extreme nuts who were maybe wrong most of the time, but insisting that we needed to fear the next pandemic, wouldn't we have been better off with that kind of portfolio? And thus, we don't actually want more accuracy from our forecasters. Well, in some sense, we already did have that portfolio. It was a mainstream position among epidemiologists for the last 20 years or so that you had a recipe for a disaster. David Epstein, the guy who recently wrote range, a very interesting guy, you may have had him on your show, I don't know. But he recently quoted himself from a 2007 newsletter that he wrote. And he said something like, the presence of a large reservoir of SARS-like viruses among horseshoe bats, combined with the culture of eating exotic meats, is a time bomb. But what's the problem? And that was in microbiology books. In the first decade of the 21st century, it was common knowledge. And indeed, even before SARS-1, even before the first SARS outbreak, some epidemiologists prefer to call COVID-19 SARS-2. But even before SARS-1, epidemiologists were acutely aware of this. So it's not as though we didn't have it in our portfolio. We did. So those forecasters maybe weren't entertaining enough? Isn't then the margin we want to work on to make our better forecasters more entertaining and not more accurate? Yes, no? Well, the signal-to-noise ratio isn't going to be great. I can assure you of that because there are plenty of people who are naturally more entertaining than epidemiologists. But maybe the whole portfolio needs to be more vivid, rather than trying to fine-tune the accuracy of particular parts of it, right? Well, you have lots of people competing in the marketplace of ideas for attention. And that's a hard competition for scientists to beat. What do you think of the argument that science only exists at all, because most scientists are overconfident? That if they were rational Bayseans, they would just latch on to the opinions of the smartest and best-trained people before them. That there's only progress precisely because people are making forecasting mistakes. Right. So you hear a version to that argument, I suppose, on Wall Street as well. Sure. That's a good deal of truth to it. I certainly have been guilty of overconfidence at many junctures in my career of thinking I'm going to be able to take on things that looked impossible and often turned out to be impossible. Most projects that most scientists embark on, I think, don't succeed. It does take a certain amount of quasi-irrational persistence. As with Columbus, right? Or the founding of the United States, arguably was irrational to break away from the British Empire, right? Which was doing pretty well back then. It was a... It seemed like a risk-seeking move. But say I set up an alternate research program and I saw it to take forecasters and, A, make them more entertaining. And, B, maybe I'd give them uppers so they were more overconfident. I mean, would that do the world good? Well, you could certainly have induced the epidemiologists who are worried about the horseshoe bats in central China or other possible sources of zoonotic viruses. You could certainly have induced them to pump up their probabilities. And you would, of course, started to run into a problem of crying wolf. But they'd been saying there's a 30%, 40%, 50% chance of a viral leap into human beings each year. And it didn't happen, didn't happen, didn't happen. People would... You'd get a crying wolf effect, right? Right. How do you think about financial markets in relation to your work on super predictors? Are financial markets, in essence, super predictors to begin with? Or can super predictors on average beat financial markets? Oh, boy. Well, you know, we play with prediction markets in the work with the intelligence community. Going all the way back to Admiral Poindexter in the original DARPA effort to launch prediction markets inside the intelligence community, I think your colleague, Robin Hansen, was involved with some of that work around 2003, 2004 in that range. We've also been working with the prediction markets in parallel with forecasting tournaments. And there are pros and cons to each method of eliciting judgments. But the IC, the intelligence community, doesn't let us turn those markets, make those markets deep and liquid the way they are on Wall Street. People are essentially competing for reputational points the way they are in forecasting tournaments. The monetary prices are either small or non-existent. So most economists, I think, would not consider that to be a very robust test of the efficacy of prediction markets. And when prediction markets fall short, and they do typically not perform quite as well as forecasting tournaments, when they fall short, it's hardly a decisive rebuke of the market mechanism for eliciting forecasts. There are lots of very powerful institutional actors like Goldman Sachs and so forth that are continually trying to do exactly what you describe. It's an ongoing process. I don't need to give an economist. I mean, there are implicit prediction markets in, say, coronavirus, say, prices of airline stocks, right? And those are very liquid. They've been very thickly traded lately. They have indeed. If you took your 10 best super forecasters and brought them into the hedge fund people at Goldman Sachs and you all sat down together, who would be teaching whom? It's an interesting experiment. The person, OK, my project manager from the first set of forecasting tournaments, Terry Murray, founded a company, Good Judgment Incorporated, which does things like that. So that's a proprietary venture. And we probably want to talk to Terry about how successful or not successful they've been in doing that. I think they've had some success. I think it's extremely hard to do that. It's non-trivial. I think there's a good deal of similarity in the cognitive ability, cognitive style profiles of super forecasters and the kinds of people you see on the staffs at Goldman Sachs. And it would be a tight race. What about the sports betting market? Do you think there are inefficiencies in that because they don't have enough super forecasters? I'm not an expert on sports betting. I'm better to talk to Nate Silver about that. But let me put the question more generally. There are many markets out there which predict something. Sports betting markets are simply the most obviously most explicit about prediction. So if there was something the world didn't know about prediction already, those markets should be inefficient. Yes or no? Why would you think the answer would be yes? Well, let's say that people favored the home team too much. So too many people might bet on the New York teams, the Los Angeles teams, and then the odds would be skewed. So if there's a bias in people without super forecasting techniques, we would expect sports odds to somehow be off or at least they would have been off before your work was published. Well, arbitrage predated super forecasting. Well, but isn't arbitrage itself super forecasting, right? People are arbitrage in a form on the basis of some set of information. I think that's fair. So all these markets out there, do you think they are without you already the best available super forecasters? The term best is a term I'm just not comfortable with. I rather doubt that they're at the optimal forecasting frontier at the moment. They're often probably fairly close. And is there a room to incentivize people to outpredict the market? Well, that's one of those paradoxes that economists have written about, right? When people predict out to many decimal places, do you think that's absurd or do you think it's useful? What's the optimal level of granularity for different categories of forecasts? I think for the kinds of things we were looking at in the original forecasting tournaments with geopolitical events like how long the Syrian civil war would last or what would Russia do in the Eastern Ukraine or things of that sort. Yes, it would be absurd to go to three or four decimals points. Originally, the National Intelligence Council which synthesizes a lot of intelligence analysis. Originally, they only distinguished five degrees of uncertainty and they didn't put numbers on it. More recently, they've moved to seven degrees of uncertainty and they do put numerical ranges on. So somewhat likely represents a certain probability range. Now, in our work, we've explored how granular it's been the best forecasters are. Doing various rounding experiments where we round their forecast off to the nearest tent, that kind of thing. And see whether, if it's pseudo-precision, if when they move from 0.6 to 0.65, for example, if on average that doesn't improve their accuracy, we would conclude that they can't achieve that level of granularity. Our best statistical estimates are that our forecasters for the types of questions the intelligence community often poses can distinguish between 10 and 15 degrees of uncertainty, which is considerably more than the seven, they think they can now a lot more than the five, they thought they used to be able to distinguish, but how useful is it to be able to distinguish varying degrees of uncertainty is going to hinge on the kind of game you're playing. If it's poker, you might well be on someone who's very adept at distinguishing things to say three decimal points. If you could take just a bit of time away from your research and play in your own tournaments, are you as good as your own best super forecasters? I don't think so. I don't think I have the patience or the temperament for doing it. I did give it a try in the second year of the first set of forecasting tournaments back in 2012. And I monitored the aggregates. We had an aggregation algorithm that was performing very well at the time and it was outperforming 99.8% of the forecasters from whom the composite was derived. So if I simply had predicted what the composite said at each point in time in that tournament, I would have been a super, super forecaster. I would have been better than 99.8% of the super forecasters. So even though I knew that it was unlikely that I could outperform the composite, I did research some questions where I thought the composite was excessively aggressive. And I tried to second guess it. And the net result of my efforts, instead of finishing in the top 0.02% or whatever, I think I finished in the middle of the super forecaster pack. So that doesn't mean I'm a super forecaster, just means that when I tried to make forecasts better than the composite, I degraded the accuracy significantly. But what do you think is the kind of patience you're lacking? If I look at your career, you've been working on these databases on this topic for what, over 30 years. That's incredible patience, right? More patience than most of your super forecasters have shown. So is there some disaggregated notion of patience where they have it and you don't? Yeah, they have a skill set. I mean, in the most recent tournaments we've been working on with them, this becomes even more evident that their willingness to delve into the details of really pretty obscure problems for very minimal compensation is quite extraordinary. They are intrinsically cognitively motivated in a way that is quite remarkable. How am I different from that? I guess I have a little bit of attention deficit disorder and my attention tends to roam. So I've not just worked on forecasting tournaments. I mean, I've been fairly persistent in pursuing this topic since the mid-1980s and even before Gorbachev became General Party Secretary, I was doing a little bit of this. But I've been doing a lot of other things as well on the side. And so my attention tends to roam. I'm interested in taboo trade-offs. I'm interested in accountability. There are various things I've studied that don't quite fall in this room. Doesn't that make you more of a fox though? You know something about many different areas. I could ask you about anti-bellum American discourse before the Civil War and you would know who had the smart arguments and who didn't, right? Well, I would know who has arguments to take the more integratively complex forms on the one hand, on the other hand, and then synthesis, whether you want to consider those arguments smarter or not is another matter. But yes, I suppose that's fair. I mean, I've always resonated a little more to the foxes and to the hedgehogs, but I think when you look at the great achievements in science, they often come from hedgehogs. If you look today at the ongoing debates about coronavirus and what will happen, and I mean now the debates amongst the smart people, the people you respect, what is the mistake you see them making? The biggest mistake. Oh, you want me to be an amateur epidemiologist here, right? No, mistake and reasoning. You don't have to give your numerical estimate. Procedurally, what are they not getting right? Well, is it a mistake if you're a public health expert who feels that one mistake is much worse than the other? It's much better to overestimate the threat of the virus than to underestimate it because you have to influence public opinion and public behavior. There you're not forecasting, you're engaged in manipulation, social influence. So, I mean, this comes back to your original question but what do we want from our forecasters? And accuracy is only one of the things we want from them. We look to forecasters for a lot of things to inspire confidence, to inspire fear, and so forth. If we're trying to estimate how much people cut back on their risk-taking behavior because they're afraid of the virus, are the group of people best suited to do that epidemiologists, some other social scientists or your super forecasters, maybe even economists, right? We study elasticities. Why should it be the epidemiologists? I think you'd want an interdisciplinary team. I think there's a, I mean, diversity is one of these words that's been reduced to a cliche but I think we have found in our work that cognitive diversity helps. And it helps in certain quite well-defined ways. If you wanna create a composite that out predicts the vast majority of the super forecasters, a good way to do it is not only to take the most recent forecast of the best forecasters in a domain, but it's also to extremize that forecast to the degree that people who normally disagree agree with each other. And when you have convergence among diverse observers, that's a signal that the weighted average composite is probably too conservative and you should extremize. Does the team, the diverse team, have a CEO, someone in charge? Not in this case, no, it's done purely statistically. But would you put someone in charge? Now maybe the person in charge would implement that statistical algorithm, right? But who's the person you would put in charge of the team? An epidemiologist, yourself, your best super forecaster, Bill Gates? That's a managerial skill and I would say, going back to one of my old dissertation advisors at Yale 40 plus years ago, Irv Janus on Groupthink, I would pick a leader who knows how to shut up and not reveal opinions at the beginning of the meeting and knows how to listen. And which group of people do you think that best describes? I think a lot of good executives have the intuition that you get more out of a team of forecasters or problem solvers. If you elicit independent judgments initially that are uncontaminated by conformity pressure and then you create an environment which ideas can be freely critiqued before lifting the veil of anonymity and letting people see who's taking which positions. Do you think having machine learning and artificial intelligence has made us much better at forecasting things right now? I mean, social events. Some kind of a range of things for sure. Sure, at the micro level, but social events. Whether there'll be a recession, how many people will die from the coronavirus? Will we settle Mars? Well, I actually just ran a forecasting tournament called Hybrid Forecasting Competition in which they pitted algorithmic approaches and human approaches and hybrid approaches against each other. And I should let IRP speak for itself about how well those programs work or don't work. I don't think there's a lot of evidence to support the claim that machine intelligence is well equipped to take on the sorts of problems that the intelligence community wanted to have answered when it runs forecasting tournaments has been running with our research team. The things like the Syrian Civil War, Russia, Ukraine, settlement of Mars. These are events for which base rates are elusive. It's not like you're screening credit card applicants for Visa, machine intelligence is going to dominate human intelligence totally. Machine intelligence dominates humans and go and chess. It may now dominate humans and poker. I don't know what the state of the art is quite there yet but it is in their StarCraft or whatever the next thing that Dennis Osabas is going to conquer. So, but no, I don't see evidence that those approaches work in the domains that we study with the intelligence community. So do you think the hybrid man machine approaches are overrated? No, I don't think it's, I think it's a matter of, it's very domain specific. It sounds like a great idea who could be against it but the devil lurks in the details and it doesn't deliver as automatically as you might hope. It's trench warfare here. Do you think the world as a whole is becoming easier to predict or harder to predict? And again, I mean social events. I'm not sure there's a definite trend one way or the other, you hear a lot of talk, a lot of claims that there are, but if you look back on the 20th century, there certainly were lots of major pockets of unpredictability, it's not clear. What if I say again, current events aside, but it seems easier to predict. There hasn't been a world war since 1945. There's been steady economic growth in most parts of the world. More peace, isn't that easier to predict? You just predict two to 4% global economic growth and you pick up a fair amount of what's happened since 1950? Indeed, so simple extrapolation algorithms are historically are hard to beat. And if you, you know, we just, we're running a COVID-19 mini forecasting experiment right now and they're proving to be hard to beat. The skill, of course, is when to alter the trend, whether to accelerate it or to decelerate it or to change direction. And do you have a personal intuition on that? I think humans have been repeatedly humbled in competitions against simple statistical algorithms. Going back to Paul Meal's famous little book on clinical versus actuarial approaches to predicting in medicine and psychiatry. So, I would say be humble. Now, there's some of your early research that if I read it properly, suggests that making people accountable leads to more evasion and self-deception on their part. Are you worried that your work with pundits by trying to make them more accountable will lead to more evasion and self-deception from them? Or how do you square early Tatlok and mid-period to late Tatlok? It's actually not too difficult in that particular case. It really depends on the type of accountability. Tournaments create a very stark monistic type of accountability in which one thing and only one thing matters and that is accuracy. And you get no points for playing to, it's for being an ideological cheerleader and pumping up the probabilities of things that your team wants to be true or downplaying the probabilities of the things your team doesn't want to be true. You take a reputational hit. So the incentives are very unusually tightly aligned to favor accuracy. That's extremely unusual in the social world. Most forms of accountability occur in organizational settings in which there are lots of distortions at work. And the rational political response for a decision maker located in most accountability matrices and organizations is to engage in some mixture or strategic attitude shifting toward the views of important others or as you put it, evasion, procrastination and so forth. But given that when a pundit is on say the evening news there's not a little box at the bottom that gives the Tatlok score of that pundit, correct? So most people don't know the actual record. So given that out there somewhere is a measure of how good or bad the pundit is, are you worried that in a sense you will make those pundits run further away from objective standards precisely because they do poorly by them? I don't know if we can make them run much further away from objective standards than they already are, but that's a very interesting point about forecasting tournaments. And I look at the kinds of people who are attracted to participate in them. At the very outset, I mean, I invited lots of big shots to participate in forecasting tournaments and they turned me down. They've repeatedly turned me down at a very interesting correspondence with William Sapphire in the 1980s about forecasting tournaments we could talk a little about later. But the app shot of this is that young people who are upwardly mobile, see forecasting tournaments as an opportunity to rise, old people like me, aging baby boomer types who occupy a relatively high status inside organizations, see forecasting tournaments as a way to lose. If I'm a senior analyst inside an intelligence agency and I'm on the National Intelligence Council and I'm an expert on China and the go-to guy for the president on China and some upstart R&D operation called IARFA says, hey, we're gonna run for these forecasting tournaments in which we assess how well the analytic community can put probabilities in which Xi Jinping is going to do next. And I'll be on a level playing field competing against 25-year-olds, 65-year-old. How am I likely to react to this proposal, to this new method of doing business? It doesn't take a lot of empathy or bureaucratic imagination to suppose I'm gonna try to nix this thing. Which nation's government in the world do you think listens to you the most? You know, I... You may not know, right? I might actually. Can I say? You know, I suppose the most prominent political fan I have at the moment is probably one of the senior advisers to Boris Johnson, Dominic Cummings, who recently caused a bit of a stir in the UK by appointing a super forecaster who had written some blogs that people interpreted as misogynist or racist or fascist or eugenicist or I don't know, some mixture of all those things. But I think his name was Andrew Sibisky who's a young man, 24, 25. The kind of person, young people are attracted, as I said before, to forecasting programs. It's a fast track toward upward mobility. You have all the high status people making vague, verbate forecasts. People like me. But do you think Cummings actually is influenced by you? Because as I understand what he's doing, correctly or not, he thinks he knows a bunch of things that other people do not. And that seems somewhat non-petlockian, right? So maybe you're part of his portfolio of ideological armor. Maybe he's actually very non-petlockian and it's the people in Singapore who are your true fans. Well, there are some people in Singapore too. And that's an interesting place. You should mention that. Interesting you're picking that one up. But Mr. Cummings and Mr. Gove, separately brought up my work at various points and during the Brexit debate. And Michael Gove, at least in the UK famously said that Britain had had enough of experts. I don't know if you remember that. Of course, particular quote. And he was thinking of, well, he even invoked as support at least for that position, expert political judgment, in which portions of that book compare subject matter experts to minimalist statistical baselines, like extrapolate. Can you predict simple extrapolation algorithms? And the answer was often no. So, Gove was raising the point that, where do these guys get off making these confident predictions about the consequences of Brexit when the best empirical evidence would suggest they're probably not materially more accurate than simple extrapolation algorithms. So that was brought up. It was brought up for a political reason. He had a political point to make. And I think this, and Dominic Cummings had a political point to make as well. He fears that I think parts of the civil service are hostile toward Brexit and want to undermine the Boris Johnson administration objectives. Now, I think this comes to something deeper now. It's not just about the UK. It's about the intellectual fissure that exists between social science and conservatives. That most social scientists are liberal and conservatives are wary of advice from social scientists. I think we may have partly paid some price for that in this epidemiological in this, in the, to put it kindly, the slowness of the Trump administration response to COVID-19. Yes, now you brought up Britain. If we look back at speeches in the British House of Commons, who is giving the most cognitively complex speeches? You know, Bob Putnam collected those data that I reported that study in 1984, but Bob Putnam collected the original data and he reported it in a book, Beliefs of Politicians, published in the 1970s. And I think those data were based on interviews with members of the British House of Commons. They were not speeches. They were interviews, confidential interviews that Bob Putnam got access to and put a lot of work into obtaining. And he shared them with me and I used them as grist for a small research program. I was running in the 1980s on cognitive style and political ideology, trying to tease apart the rigidity of the right versus the ideologue hypothesis. And who sounds the smartest from that period? In that period, it was a mixture of moderate labor, moderate conservative. It was the centrist that did better. It was slightly left shifted. Do you think we can draw any inferences from that about politics? Should we have more faith in the people who sound smarter or not at all? Well, I think that particular measure in integrative complexity is, I think it does have some correlation with forecasting accuracy, but I think you're picking up something more than just forecasting accuracy. You're picking up what I call value pluralism. You're picking up a tendency to endorse values that are often in conflict with each other. So the more frequently that you, as a political thinker, confront cognitive disness between your values, your value orientations, the more pressure you are to engage in integratively complex synthetic thinking. When your values are more lopsided, it's easier to engage in what we call simpler modes of cognitive disness reduction, like denial or bolstering and spreading of the alternative. So we downplay one value, push up the other value and make your life stress-free. But some value, some ideological positions at some points in history, require more tolerance for disness and some people are more inclined to fill those roles. And you think those politicians are also likely to be better forecasters? You know, we're not talking about huge effect sizes here. I think integrative complexity. I think that fluid intelligence is probably a more powerful predictor, but a combination of fluid intelligence and integrative complexity, I think does boost forecasting accuracy, yes. And if you were running the CIA, those people who are pluralistic in the manner you just outlined, would you promote them more rapidly? Now, of course, the CIA, bear in mind, is by statute supposed to be value-neutral. It's not there. They're supposed to be feeding impartial, apolitical advice. Just the fact- Oh, I believe that, right? Well, that is supposed to be the division of labor here. Now, is it, and forecasting tournaments are, I think one reason they may be interested in forecasting tournaments is because forecasting tournaments incentivize people to do one thing and only one thing, and that's accuracy. You don't get points for skewing your judgments toward your favorite cause. You take a hit on the long-term by doing that. If you were in charge of the CIA and had a free hand, how would you reform it? Oh, boy. Well, there's a long history to efforts to reform the CIA. Going back to, well, it was founded in 1947, and there have been various efforts since then. People have been unhappy with the CIA for many reasons over time, you know, Vietnam being a big one, but there are lots of other reasons that people have expressed unhappiness. Both liberals and conservatives have at various points been unhappy with the performance of the intelligence community. In 2001, it came to a kind of a crisis point, I think, and then there was a commission to reform intelligence analysis. And after the WMD fiasco in Iraq, that the pressure grew even more. One reason why we're even talking right now today is that the intelligence community was forced, essentially, by the recommendations of the reform commission to take keeping score more seriously, to take training for accuracy and monitoring of accuracy more seriously. And I think that that's when they created the Intelligence Advanced Research Projects Activity, which is the R&D branch housed within the office of the Director of National Intelligence. And its job is to support innovative research that will improve the quality of intelligence analysis where that can be defined in various ways, but accuracy is certainly one of the very important component of that. But it's not just, but it's not accuracy with a liberal skew or a conservative skew, it's supposed to be just plain, just the facts, ma'am, accuracy. But what questions they choose to predict, right? That reflects values, obviously. I'm sorry. Which questions they choose to predict that reflects values. Whenever bureaucracy tells me they're value-free, I start getting more suspicious very quickly, right? I never believe them. It might be okay for them not to be value-free, but the cynical response is the correct one here. Well, indeed, it's a really old expression. There's no view from nowhere. There is no such thing as pure value neutrality. That doesn't mean it's not something worth aspiring to, but you're what you're right, the value's off. Even if you had a perfectly objective forecast in turn of in-system, if you had people generating the questions and promoting a political agenda, you could skew the results. I think that's one of your points, right? Now, in the middle of all these discourses, we have a segment called overrated versus underrated and I'll toss out a few names, ideas, and you tell me if you think they're overrated or underrated. How's that? Okay. Philadelphia, the city of Philadelphia, overrated or underrated. I got endless grief when my wife and I decided to leave Berkeley and move to Philadelphia. They, people thought that we were borderline insane, but we left for very personal reasons, and without going into what those were, I would say Philadelphia's been a moderately pleasant surprise. It's a city that has many, many, many problems, but it's not as bad as the people in Northern California thought it was. Tolstoy. You know, I haven't, I read a little bit of Tolstoy and I've seen a number of films. And I know some of the short-hand versions, and I know that Isaiah Berlin had a hell of a time classifying Tolstoy as a Hedgehog or a Fox. But something I'll pass on now. John Cleese, he's commented on you. You're allowed to comment on him. Right. Well, he gave me, I had a wonderful time as a kid watching Monty Python and also Faulty Towers. So I really enjoyed his comedy. I haven't followed him since then. But when I was younger, I thought he was absolutely hilarious and brilliant. And I appreciate the flattering things he said about my work. The television show, The Sopranos. I fell for it. And I loved it. James Gandolfini, I fell in love with his performances. And then when he, you know, one of the last things he did is he played Leon Panetta in Zero Dark 30. And there he is, you know, listening forecasts from people in the CIA about whether Osama bin Laden is in that compound and about of that. Is he there or isn't he effing there? Great, wonderful. I did never do him, but I think very highly of his work. The threat of terrorism. Do we overrated or underrated in the United States? That's a very difficult question because of the tail risk aspect to it. If you looked at the number of people who died from terrorism versus other causes, it would seem that the amount of money we spend on suppressing terrorism would be disproportionate. But the tail risk complicates that a lot. What is your favorite movie? I don't have a hierarchy like that. I'm sorry. Nothing comes to mind. What's the movie you've seen the greatest number of times? That you can count. I don't know if I watch movies often over and over. I did see myself coming back just very recently, last week in a quarantine period, to Westworld. It's not a movie. It's a series, of course, but I think that is quite, I think the first two seasons are quite brilliant. On historical counterfactuals, by what year do you think the ascent of the West was more or less inevitable? Well, I have inside information here. We did a survey of some very prominent historians. We reported it in that book on making the West. We reported part of it anyway, also in an article on American political science review. I think that if you've looked at just the unweighted average of judgments for the media, and I think it was probably around 1730, 1740. And you think after that it was not very contingent? That would have been, you see, I'm not a historian of the West. I mean, what do I really know about that? I'm simply reporting the news here. And is there any great hinge of contingency that you think about looking backwards? Like, oh my goodness, if there hadn't been a Reformation or if there hadn't been a Council of Trent or what? Well, I'm a fan of Steve Pinker. I think the Enlightenment was a big deal. Because it got people thinking more rationally and more in terms of science. And that had huge spillover effects. Now, could there have been an Enlightenment in China or could the Chinese have created certain types of technologies without science? What if they hadn't scuttled their Navy in around 1400? They're all those sorts of counterfactuals. And how necessary or contingent do you think it is that we keep on thinking in Enlightenment-like terms? Is it once you're locked into it, it keeps on going? Or is it like good government that you have to renew it every generation or two? Well, I see the work I'm doing is very much in the spirit of the Enlightenment. Public, transparent standards of evidence for judging subject matter expertise. I think one of the great challenges of our time is striking the right balance between democracy and technocracy. And I think the fissures that have emerged, it's not just conservative or very skeptical of social science. And it's apparently some parts of biological science too, which is I think very unfortunate. But you see this kind of some reflexive skepticism towards science on the left as well. So, I mean, how do you manage the relation between small D Democrats and technocrats in a society in which expert guidance is increasingly crucial? What are you learning from playing the game Civilization Five or at least watching others do so? That's a good example actually of why I'm not a super forecaster, I don't play Civilization Five, but Civilization Five was one of the simulations that IRPA chose to feature in its counterfactual forecasting tournaments under the rubric of focus. And if any of your listeners are interested in signing up to be forecasters for focus, we still have one more round, round five, and we will be recruiting people. But again, it reflects my temperament. I don't have the patience for a game like Civilization Five. That requires a lot, you know, forecasting is inevitably a mixture of fluid and crystallized intelligence. And you have to invest a lot of energy into mastering a game like Civilization Five. And I suppose as people get older, they may become less likely to make those kinds of cognitive investments. I mean, it becomes more and more essential as I get older, I think, to focus on the things where I have a real comparative advantage. The best chess players are all young, right? Yes, we know. There's clear data. Yeah, I mean, it's interesting the domains in which gel prodigies emerge, music and chess and math. If we take super counterfactualists and super forecasters, do those two groups basically overlap or how do they differ? I think they have to be rather intimately connected, although disentangling this one is going to be really, really hard. And it's one of the things I do wanna dedicate a few years of my life to doing. Now, obviously, when you say someone a good counterfactualizer, some people shrug their shoulders and they say, well, how are you possibly gonna know whether you would have gotten undue the assassination of the Archduke in 1914, do you undo World War I? Undue Hitler, you undo World War II. You make Kennedy grouchier during the Cuban Missile Crisis, you trigger World War III. You've got these sorts of arguments that are essentially unresolvable. You can't rerun History, you can rerun Civilization Five, but you can't rerun History. So that makes counterfactuals a place where ideologues can retreat. They can make up the data, they can make up whatever facts they want to justify pretty much whatever, no matter how bad the war in Iraq went, you can always argue that things would have been worse to those who say it remained in power. So you have these counterfactual, and counterfactual reference points that people use in debates implicitly to make rhetorical points. So a lot of people, part of what attracted me to counterfactuals was, A, how important they are in drawing any lessons from history, B, how important they are in policy arguments, and C, how unresolvable they are. Now, one of the things I think we're hoping to do in the focus program is to develop some objective metrics for identifying people and methods of generating probabilities that produce superior counterfactual forecasts and simulated worlds in which you can rerun History and assess what the probability distributions of possible worlds are. Well, it turns out you get World War I 37% of the time, even if you undo the assassination of the Archduke and you get something like World War II, you can see where we're going. So we're hoping that one result of focus will be to help us identify people and methods that generate superior counterfactual forecasts and domains where there is a ground truth. The next task will be to connect superior performance and simulated worlds to superior performance in the actual world. And this is where things get tricky, of course, because in the actual world, we don't have the ground truth. So if I ask you a counterfactual question of the form, if NATO hadn't expanded eastward as far as it did in 2004 into the Baltics, NATO-Russia relations would be considerably friendlier than they are now. Are chess players good forecasters? But do you want to go with this for a second? Sure, I'm sorry. I thought you were stopping. No, I don't want to stop on that one. No, keep on going. We've got a counterfactual there. We can't rerun History. We don't know how relations with Russia would be if NATO hadn't gone into the Baltics. Russia would have gobbled up the Baltics maybe Russia would be friendly or feel less threatened. You have people who have more hawkish or more dovish mental models of Russia and those mental models predispose them to give you certain canned, almost ideologically reflexive answers to those counterfactuals, right? Right. But you can measure what people's beliefs are on the counterfactual and then you can measure people's beliefs about conditional forecasts that are logically connected to the counterfactuals in kind of a Bayesian inference network. So you can measure, you know, if I know that you think that the Russians would be every bit as nasty and snarly, even if we hadn't moved into the Baltics, they might even be nastier. It's probably a fair bet that you're also likely to think it's a good idea to increase arms sales to the Ukraine, ratchet up sanctions on Putin's cronies and so forth. So we can identify the counterfactual belief correlates of more or less accurate conditional forecasting. And in that sense, you can indirectly validate or invalidate, you can render more or less plausible certain counterfactual beliefs. That's the longer term objective of this research program. It's not just, we're not playing Civilization 5 for the sake of getting better at Civilization 5. The ultimate goal is to link the sophistication of counterfactual reasoning about the past to the subtlety and the accuracy of conditional forecast going into the future. Another thing you should observe, by the way, if people are becoming better counterfactual reasoners is you should observe less ideological polarization in their counterfactual beliefs. So counterfactual beliefs should become as ideologically depolarized as conditional forecasts are. This playing world of Warcraft a lot help you become a better forecaster or playing chess. I don't have any evidence bearing on either of those things. But there's a third variable problem there too. But you get used to a test, right? You know if you lose, there's very little self-deception. This is true. I mean, the people who do well in these sorts of things often like games like that. Venture capitalists, when they try to spot talent in others, do you interpret their behavior in terms of a super forecaster model? Someone like Peter Thiel, he found Mark Zuckerberg, Reed Hoffman, Elon Musk, he's a kind of super forecaster. How do you super forecast talent in other people? Well, it really helps to be working in an environment in which super bright people are not super rare. It's very, very hard to forecast, to identify talent when the base rate falls below one in a thousand, one in 10,000, one in 100,000. You're looking for a needle in the haystack. But the great advantage that venture capitalists in Silicon Valley have is that the talent pool is relatively rich. So there's quite a few flaky people that come by seeking their money, for sure. But their odds of success are significantly better than they would be if they're working from a population base rate. And of course, they can tolerate a lot of mistakes because just a few hits will pay for a lot of false positives. But some do much, much better than others, right? Mike Moritz, Peter Thiel. Right, the question is, are they doing better because they have better social networks? Are they doing better or they have better judgment? Do you think super forecasting as a technique also applies to super forecasting how people will do? I'm sorry, super forecasting as a technique for predicting the future. Does it also apply to predicting how successful people will be? Yes, I think everything we know from, you know, the overlap between super forecasting and intelligence and the overlap between intelligence and success in various, in many different professional lines of work would suggest that was almost certainly true. Who first super forecasted your own major success? I don't know what my first major success was and some people might tell us whether there has been a major success in my career, but who first saw it coming? I think probably my advisor at University of British Columbia, Peter Sudfeld, who supported me and believed in me when it didn't seem to be very many good reasons for doing so. Kind of a confused Canadian kid who was unsure whether to become a lawyer in Canada or go off to Oxford or go to graduate school in the US who tipped me toward social science in the US. And what did he see that other people had not seen? God only knows. Well, if anyone would know, it's you, right? You've lived with it for some time. I think he probably saw it. I thought I was probably bright enough to do well and he probably thought that I was contrarian and weird enough that there was some possibility of doing something distinctively well, something distinctive and different and doing well. Do you think academic advisors in general today undervalue weird students? I think there's probably a tendency in that direction, yes, because weirdness is a big word. Weirdness takes lots of forms that you and I would know. We would not be embarrassed about turning away lots of weird people. Sure. Do you think people who grow up with the second culture are better forecasters? Oh, you're thinking of my work with Karmie, Karmie Tugmore. Of course, yeah. Gosh, you really looked into my vida here. This is highly unusual. I think there does seem to be some advantage there, yeah. And where does that come from? How does it work? I think it has to ties into accountability, accountability to conflicting audiences and value pluralism. You have a richer internal dialogue. You learn to balance conflicting perspectives more. You have to be a better perspective taker. Perspective taking is a very important part of superforecasting too. How do we create more innate silvers and fill up catalogs? What should we change in the world to get more of you? Well, I think they and I are probably very different creatures. Sure, but we want more of you both, right? What should we do? You know, one thing I think that would be useful, I mean, there is this tendency for training in universities to become hyper-professionalized and compartmentalized. So I think it is harder for people who have weird interests that straddle, say, psychology and organizational science and political science and law and history. People who have weird sets of interests, it's hard for them to get traction in the current career environment. Certainly, in my home discipline of psychology, I mean, you had this kind of rampant publication inflation in which PhD students would be very lucky to get a job if they have to have a ridiculous number of publications and some of them in A-journals. It's the sort of thing we didn't really expect and 30 years ago we thought, oh, that's a tenure case. Oh, that's a junior hire. So those kinds of pressures, I think, produce a focus, an narrowing of focus. Now you say, well, yes, what you, you know, the great, I said earlier that had a lot of the great advances in science come from hedgehogs. We're producing hedgehogs on an industrial scale here. And I think there's some advantage that may make it a little bit more room for weirdo-eclectics. And the way that departments are carved up, would you change that at all? You know, a number of academics in the past, like Jim Marge, you see Irvine many decades ago tried to do something like that. And Amy Gottman, the president of the University of Pennsylvania is trying to do that with Pan Integrates Knowledge, which is the chair of professorships that we're used to among other things to hire a barber and hire a number of other people. And we are sort of floaters. You know, we're not connected to one unit. We have multiple connections. So, and that's a very hospitable work environment, in my point of view. But I don't see a lot of places doing at Jim Marge, you see Irvine experiment. Well, let's integrate all social sciences together. And Amy Gottman, PIK, Pan Integrates Knowledge kind of program. You don't see too many of them yet. It be friends against the grain. I'm not even sure it'll survive at Pan beyond Amy. I mean, the natural tendency will be for departments to want to claw back the resources. Let's say someone comes up to you and they say, Philip, I would like to be more Fox-like. I'm not enough of a Fox. What actual advice would you give them to achieve that? What should they do? Or not do? Wake up earlier in the morning, exercise more. Yeah, I'd commit career suicide, become more Fox-like. Maybe they're not an academic. They're a smart business person. How do they do this? I, you know, kind of obvious things like read a little bit more outside your field. If you're a liberal, read The Wall Street Journal. If you're a conservative, read The New York Times. Expose yourself to distant points of view. Try to cultivate some interest outside your field. Try to connect them together. I think there's an optimal distance. I mean, so for history, for example, is, you know, sounds quite very different from what I did, but I started as an experimental psychologist and history looks very different, but they can be connected because historical judgment is something that psychologists study to some degree. Psychologists are interested in hindsight and counterfactuals and so forth, so you can link the two. So I think there's an optimal distance and when you go foraging as a Fox, you probably don't want to forage, you know, way, way far away. You want to forage far enough away that it'll be stimulating, but still possible to reconnect. What kinds of people are best at adversarial collaboration? Rare people. Those are rare people also. But how do you spot really, really hard to do? Is it a personality trait, a cognitive trait? Gosh, that's a hard one. I mean, that was a thing that Danny Connem, I think coined the term when he was, you know, dealing with his various critics over time and my wife Barb actually was involved in an adversarial collaboration between the Connemann camp and the Giger-Inzer camp on the conjunction fallacy. It took at least two or three years of her life. It's a hard, it's very hard to get people to, it, well, I mean, lots of us in principle are barbarians. We believe in stating our beliefs as falsifiable hypotheses and we also, most of us believe our beliefs are probabilistic, so we're kind of somewhat Bayesian. But that's lip service. There's what we believe in our, there's our formal set of, there's our epistemological formal self-concept, which is kind of noble, falsificationist and probabilistic. And then there's how we actually behave when our egos are at stake in particular controversies. And those things are quite different. I think Danny Connemann, who's not known as an optimist, you know, it did propose, it did propose it. It sounds like an optimistic idea, but I think he's not all that optimistic about what it can achieve on close inspection. My efforts at adversarial collaboration have not been all that successful. I'd like to jumpstart a few of them again, but it's very hard to find the right dance partners. Let's try a question from the realm of the everyday in the mundane. If I go around and I look at Mexican restaurants, I'm very good at predicting which ones have excellent tacos. What are you good at predicting? I'm not good at predicting your questions. What am I good at predicting? I think I was pretty good at anticipating the fragility of a lot of micro-social science knowledge prior to the replication crisis erupting. No, I mean everyday life. Oh, okay, so not in my... Not social events, not social science. What in your life are you good at predicting? When you're going to get tired and want to go to bed at night or when the dog wants to eat, what is it? Well, we have a pet-free existence and our lives are actually pretty, pretty simple. We get subscribed to that old adage. You should be boring and stayed in your life so you can be violent and creative in your work. We have kind of a routine here. So it's highly predictable. And now, in the quarantine days, I mean, it used to be pre-quarantine, we said, oh, we're going to go to Europe, we're going to go here there. There were these little points of unpredictability that spiced up life, but those do not exist right now. And two more questions to close. First, what can you tell us about your next project? Well, I think the next project is the one that I mentioned earlier. It's linking historical counterfactual reasoning with conditional forecasting. I think it'll be the second phase of the focus research tournaments. I think that counterfactual reasoning has for too long been the last refuge of ideological scoundrels. Insofar as we can improve the standards of evidence, improve in judging counterfactual claims as well as conditional forecasts and linking the two, I think it was a potential for improving the quality of debates among interested parties. And finally, what should a super forecaster predict about the future course of your influence? Oh, to be very cautious, because we're running against the grain, we're running against the psychological grain, we're running against human nature, we're running against the sociological grain. But the enlightenment is stable, you tell us, right? So if it keeps on accumulating and growing, your influence should be enormous. Find your other presuppositions. Well, there's a lot of cognitive resistance to treating one's beliefs as falsifiable, testable, falsifiable probabilistic propositions. People naturally gravitate toward thinking of their beliefs as igoo-defining, quasi-sacred possessions. That's one major obstacle. Then you have the existing status hierarchies. You have subject matter experts who are entrenched to have influence. Why would they wanna participate in exercises in which the best possible outcome is a tie? Which they reaffirm that they deserve the status that they already have. So that's not, psychological, sociological resistance. It's interesting, sociologists and economists have kind of different reactions to forecasting tournaments. The sociologists, the reactants, why would anyone be naive enough to think that anyone would wanna have a forecasting tournament in an organization? They're status disruptive, right? Yes. And economists would say, well, if these things are so great, how come they're not everywhere? And with that, Phillip Tetlock, thank you very much. It's been a pleasure. Take care.