 Wow, this is indeed a full house. I know it's a pun, but there's a lot of puns here. Those of you who have been to MOMAT that ever looked at our exhibit names know that we're very fond of puns. And if you have never been to MOMAT before, I hope this is your first time, but not your last. I hope you'll come back when our exhibits are open to see what we have to offer. Just to let you know, there are a lot of exciting things coming up at MOMAT. If you like tonight's talk, please sign up to be on our mailing list. Email info at MOMAT.org. You'll hear about our upcoming lecture with artist and mathematician Frank Farris. You'll hear about our book clubs, our game nights, our programs for teens and tweens, for adults and kids. We hope everybody will come back and join us. But of course tonight you're all here to see someone other than me and other than MOMAT. I am so delighted that we have Annie Duke with us tonight. Annie has been a friend for a long time. We first met before the museum was open and we did some programs together and I'm delighted to have her back now. You probably all know that she was an award-winning championship poker player. You may not know that she retired in 2012. She's now not just a full-time consultant, but also as you can see on the screen behind me, she's a best-selling author. And we changed that screen not too long ago because her book just hit the best-seller list. So that's really exciting. For those of you who have read her book, it won't surprise you to know that she studied cognitive psychology at the University of Pennsylvania. She was on her way to a PhD when the poker bug bit her and she went in a different direction. But her book really does a masterful job of combining poker and cognitive psychology. If you haven't read it, we do have copies here. I hope you'll pick it up. She'll be signing. I'm also delighted that we have Gary Marcus with us tonight. He's an NYU professor, a best-selling author, and the founder and CEO of a company called Geometric Intelligence, which certainly sounds like a company that Moments should be fond of. And I know you're going to enjoy hearing from Gary as well as Annie. So without further ado, please welcome Gary and Annie. In the spirit of the venue, I thought I would tell you a short story about parallel lines and actually parallel lives. Why it is that I agreed to do this the second time that I interviewed Annie in a month, which I don't think I've ever done that for anybody. Probably nobody's ever asked me a second time. So many complications there. But I think of us, Annie and I, to some extent, having parallel lives that then completely diverged and then suddenly we're back on track and parallel. So we met in around 1990 or 1991, I'm not sure which. At Stanford. At Stanford. Neither of us were Stanford students, but there was a Stanford conference. And we were both students of some of the premier researchers in language acquisition, how kids acquire language. And it was a conference on that. And some of my older friends, I was very young. I wasn't old enough to drink, I think at this point. I had snuck into graduate school. Some of my older friends had to meet my friend, Annie. And so we did. And we just met each other at that one conference, never saw each other again. But she left a huge impression on me. Most people I meet once, I would not remember again. Most people I meet many times. But she left a huge impression on me. And then I noticed all of a sudden she had left the field that I was in behind. So here we were trying to get our PhDs in language acquisition. And she abandoned ship to become a poker player. And then I watched from afar, as she did quite well, playing poker, which was no surprise to me, because she was obviously, you know, a star in the making in language acquisition. So I wasn't surprised that she should become a star in something else. And then we lost, I mean, I didn't see her for a long time, even though I was watching her. And then a mutual friend introduced us a few years ago, when she was starting to write this very book, Thinking in Bets. And funny thing is, I had written a book about human cognitive errors and how people think and so it's worth called Cluj. It was not on the bestseller list, although it was on the bestseller list. So we both written books about human cognitive error. She was writing one then and I had finished one a few years ago. Now I am pleased to say that she is joining the bestseller list company of people writing about human cognition. And even though she went off to play professional poker for a long time, she shared I think my interests in human cognitive psychology the whole time, then she retired as I sort of have my own way, but maybe we'll get there, maybe we won't. And started this whole new career, again, as a cognitive psychologist. So she returned to the fold from which we were both raised. So it's kind of exciting to to run into her periodically. Well, I think that what I would say is that sort of discovering through this process of in 2002, I started actually speaking on on this topic, which is how does this game of poker kind of inform the way we think about human cognition and started figuring out at that point and much more deeply, even in the process of this book that I didn't really come back to the fold so much as never left it. I think that when I first started playing, I thought I was leaving the fold. Well, you're doing applied cognitive psychology. Yeah, that's right. I started doing applied cognitive psychology. But so, you know, what I realized is I thought I was really taking this big sort of left turn away from studying human cognition. And I figured out, you know, relatively fast that I just sort of had landed myself in a different kind of laboratory to study it. So the title of I never left. I know. So the title of the book is thinking in bets. And very close to the beginning, you talk about a bet that most people think was one of the worst in sports history. Can you tell us about what what was at least perceived as one of the worst bets in sports history? Yeah, so it's 2015. And the Seahawks are on the one yard line of the New England Patriots. How many people remember this play? How many people know what's for you? Football. It's the Super Bowl. That's awesome. It's the Super Bowl 2015. The Seattle Seahawks are on the one yard line of the New England Patriots. It is second down. The Seahawks have one time out. And there are 26 seconds left on the clock. A long story short, Pete Carroll calls for a pass play there. The Seahawks have a guy named Marshawn Lynch, who is you know, one of the best running backs in the history of football. Everybody's expecting Pete Carroll just to hand the ball off to Marshawn Lynch. He doesn't. Russell Wilson passes the ball. And it's intercepted by Malcolm Butler in the end zone to end the game. The Seahawks were down by four. Obviously, if they score a touchdown, they're most mostly going to win. I mean, I guess occasionally the Patriots could get back, but they don't have very much time to do it. Chris Collinsworth, while calling the game, pretty confidently says this is the worst play in Super Bowl history. The next day, the headlines seem to really agree with Chris Collinsworth. The headlines were really brutal to Pete Carroll. Just over and over again, it was just, you know, there was one article that called him an idiot. There was seem to be the argument was not like was it a bad call? It was just was it the worst call in Super Bowl history or the worst call in football history period. So I opened the book with this because there seemed to be general consensus around the horror of this play. And I asked people to do this particular thought experiment, which I'm going to ask you guys to take a second with. Imagine that Pete Carroll passes the ball. Has Russell Wilson throw. Those who do not know the sport can exempt themselves from this. Yes. But actually, even if you don't know football, you could have I think that you can probably get this slide from it. So he has Russell Wilson pass the ball. And let's say that the Seahawks catch it in the end zone for a game winning touchdown. What do you think the headlines sort of look like the next day? Most brilliant play ever. Most brilliant play ever. Yeah, exactly. Out Belecek Belecek. Much like people talked about that Peterson this year, with the Eagles, right? Like, oh, the Philly special. He's such a genius. Well, that is that happened to work out. So I'm really, I'm really starting this book with the concept of the quality of Pete Carroll's decision should actually have nothing to do with the outcome of the decision. The quality of the decision exists actually on its own outside of however it turned out. And the fact that our feelings about it change so much, depending on whether the ball is intercepted or whether the ball is caught for a touchdown should tell you something about the foibles of human cognition, which is that we really too tightly tie the quality of the result of the decision with the quality of the actual decision itself. And this is something in poker that we call resulting in cognitive science, you would call it outcome bias. And it's a really, really big problem because there's actually a lot of uncertainty, just so that you know that there's one fact that really kind of digs deep into this call by Pete Carroll, which is that the rate of interception there hovers around one ish percent. So it's a very, very tiny part of the possible futures that could occur from this past. So I think we can agree it was probably pretty unlucky that it got intercepted there, given that it doesn't happen very often, but 99% of the time other things happen. And yet, people really universally, you know, just assaulted Pete Carroll for the decision, because of this resulting problem. So that I sort of start the book with that thought experiment. And that's really where the book goes from there is really kind of thinking about how we get messed up by uncertainty. So another theme in the book throughout actually is future and past self, you have a great anecdote there, if you can reconstruct of course, about Jerry Seinfeld and bedtime. Can you tell us that story? Yeah, sure. So one of the things that I really talk about in this book that I ask is, so we know that there's all these different ways that our cognition is not necessarily the most rational. So resulting is one of those, right, that we really look at, you know, what's the quality of the decision? Well, I'm just going to look at the quality of the outcome and go from there. There's all sorts of other types of biases that are incredibly hard to overcome. Confirmation bias, which some of you may have heard of, which is you look for information that confirms beliefs that you already have. And you kind of ignore information that doesn't. There's something called motivated reasoning, which is a way in which we reason toward conclusions that we want to get to. A good one, self serving bias. If something bad happens to me, like I get in a car accident, it's clearly not my fault. But if something good happens to me, I'm a genius. So we have all these cognitive biases. And the question is, how do we solve for these kinds of things? Right? How do we get in and try to figure out how do we get better at this stuff? And the argument that I make in the book is that, well, you can't, it's kind of hard to do on your own, that you need to bring in help. Because on our own, it's like we're just sort of like, you know, the heuristics and biases are pretty strong. So when I say help, sometimes I mean help from a different version of you, which is where I get to Jerry Seinfeld. So I kind of think about you have these different versions of you, there's like past Gary, and future Gary, and present Gary. So one of the ways that we can really help ourselves become more rational is to think about how can we recruit different versions of ourselves. And Jerry Seinfeld, I think, actually gives one of the best examples of the problem here. And he does a bit, you guys can tell me if anybody's Seinfeld fans in here, which is Morning Jerry and Night Jerry. So it goes like this, you know, Morning Jerry really screws Night Jerry over all the time, because, sorry, the other way around, Night Jerry always screws Morning Jerry around over all the time, because Morning Jerry is like out, he's having a great time. He's having fun. He wants to stay up late. You know, what about Morning Jerry at you tell with Morning Jerry, I'm going to have fun. And then Night Jerry, you know, Morning Jerry wakes up, it's like, Oh, damn you, Night Jerry, because Morning Jerry has had no sleep and can't function the next day. So this is kind of a fundamental problem is that in general, we tend to be the way that we tend to be processing information is in the best interest of this moment of who we are now. So if we go back to that thing about self serving bias, we kind of can kind of see how that is, right? If I think all bad things aren't my fault. And all good things are to my credit, that's probably going to inhibit my ability to learn. Right? So I have this future goal, like future Annie wants to be better at decision making and future Annie wants to have better results and future Annie wants to become an expert. But in order to do that, I need to learn from the stuff that's happening right now for present Annie. And one of the things that's going to get in the way of me learning is if when I have bad results, I kind of ignore them. So I just think there's luck and I pretend like there's nothing to learn from them. And when I have good results, I'm just always reinforcing those behaviors. What happens when you do that? Well, for example, you might start thinking that you drive better when you're drunk. Hopefully none of you actually come to that conclusion. But that's how that kind of thinking kind of gets you because I'm taking care of the way that I feel now. It hurts to think that a bad outcome is because of something I did. It feels good to think that good outcomes are because of something. And so in order to take care of the way I feel now, I kind of screw over morning Annie. I make it harder for morning Annie to do better and to think better. So how do we sort of get these different versions of yourself to talk to each other? And that's really where the Jerry Seinfeld thing comes in. How do we get morning Jerry to tap night Jerry on the shoulder and be like, hey, night Jerry, could you please go to bed? So I'll just give you one small strategy that you can use to do this. And it's called the Ulysses contract. How many people here have heard of Ulysses contract? Has anyone heard a few people? How many have heard of Ulysses? Right. So Ulysses is the Roman name for Odysseus. Why did they call it an Odysseus contract? I don't know. That's so weird, like because it's his more famous name. Maybe they were James Joyce. I don't know. What? We make that literature even more messy than it already is. It's a story about the Iliad, right? Yeah. He asked his crew members to tie him down so the sirens couldn't. That's right, that was his contract. That was his contract, exactly. So you may not, I don't know if you could have heard him, but right. So Odysseus is trying to get home to Penelope. He knows there's going to be some trials and tribulations along the way. There's going to be some tests. One of them is that he's going to have to fail by the island of the sirens. The sirens stand on the shore. They sing a song which is irresistible to any man's ears. And any man who hears this song will steer toward the shore, toward the siren song, which is a rocky shoal, which leads to certain death because the ship will break apart, hitting the rocks and you'll die. So Ulysses knows that this is coming. So we could think about this, right? This is like Ulysses thinking about, oh, future Ulysses is going to have a problem. So how can I help future Ulysses with this problem? So basically he binds future Ulysses hands, literally. He says to his crew, take my hands, bind them behind the mask. That way when I hear the song, I won't actually be able to grab the wheel of the steering wheel of the ship and actually steer toward the shore. Incidentally, he also tells his crew to put wax in their ears so they can't even hear the song. So there's all sorts of ways that you can implement Ulysses contracts. A very simple way, speaking of driving drunk, is to take a ride-sharing service when you're going to go to a bar. Because you can recognize, like I can recognize that when I go to a bar, if I've had a couple of drinks, I probably won't be my most rational self in trying to figure out whether I should be driving or not. So I can literally just take that decision out of my hands. So that's a version of me helping myself be more rational by recruiting different versions of me to kind of get into the mix. By getting, you know, morning Jerry to tap night Jerry on the shoulder and be like, hey, I'm going to exist and fundamentally we're the same person, so maybe you should take a little care of me. So there's all sorts of ways that you can implement those. Like another one would be if you decide that you don't want to eat sugar anymore, like throw all the ice cream out of your house. Now does that mean that you can't get ice cream? Well, of course not, because you could, you know, get it delivered or you could go out at midnight to go get your ice cream. But at least it creates a barrier to doing so, which makes it actually harder to act on it, which means you'll do it less. It's a more modest barrier here in New York because the ice cream places are so close by. You still have to like, it's midnight, you still got to like get your stuff on and go downstairs and it can get you out of your emotional mind into the the front part of your brain a little bit better. So speaking of taking decisions out of one's hands, how many people here know the famous marshmallow task where kids are told you can have one marshmallow now or two if you wait. If you YouTube for it, there's a classic video where you watch kids struggling to to, you know, wait for that second marshmallow. It's actually a version of Ulysses contract and some of the kids will sit there squirming in different ways and some of them will actually sit on their hands. I have occasionally stolen that page myself and I'm at some big dinner and I can't really focus and there's dessert that I don't really want to eat for my health and I'll actually sit on my hands sort of Ulysses contracts out and halfway towards, you know, what the, or all the way towards what the four-year-olds are doing. It kind of works for me. Yeah, and like the videos are so great, like they'll turn their chair around so they can't see it. That's another thing. I'm not looking. It's not there. So yeah, that's like about, I think it's about 15% of four-year-olds can actually wait and then there's famous study which I'd like to see replicated but which alleges anyway, well it doesn't, I mean the study is real but the sample size is small. But there's a super small sample that correlated children's ability to do this at four with something like their achievement or their SAT scores. Yeah, you know, 15 years later the claim was there was a reasonably strong positive correlation for negative and negative. The longer they could wait the better they did on their SAT. So let's call those kids better time travelers. Or having more self-control. Because there's better time. There's better time. We're going to have a smackdown about causality in this one. We will. So in that period in which you at least ostensibly left us cognitive psychologists behind and in fact we're making your living as an applied cognitive psychologist, I have always inferred that you were probably taking the literature on cognitive errors to a good effect and exploiting the cognitive errors of your opponents on occasion. Is that a reasonable inference or are you willing to tell us which ones were helpful in allowing you to win the World Series of Poker? Well so I think that in general having that knowledge about bias was really important. So the thing about poker is try to figure out what your opponent is doing that you can kind of like dig into so that you can actually win against it. So one of the things that's actually really interesting with poker is that people can have very very strong prior opinions about how you might play based on things that are superficial to you. So a simple example would be if you sit down at a table and someone is in a full-on business suit you would have expectations about how they play that are different than if they're 21 with headphones on and sunglasses and a baseball cap. So you would sort of think different things about how they might play. So one of the things that really affects people's opinion of you when you sit down to play at a table is if you're a woman. So I kind of understood confirmation bias and how much that sort of digs in and how much people will sort of process information to affirm what they already believe. And I found that when people have very strong priors about you that even in the face of data they'll very often not update very well about who you are. So they think you're a lousy player. You take all their money and they think ah she's just lucky she's a lousy player she's a girl. That's exactly right. And I didn't try very hard to convince them otherwise. So like let me give you an example. So one of the ways there's different ways that people will assess you. But one of the very common ways and this obviously is only true of a certain sector of people that you're playing against because there are lots and lots and lots of men who don't think that about women and who take you as you come and play against you as any opponent and just try to figure out how best to beat you. Those people are less fun to play against. But here's an example of sometimes they the worst thing that could ever happen to them was if a woman beat them in poker. That would be like the horror. So that would express itself in certain ways which was they really like to bluff you because bluffing in poker is when you win without the best hand. So you actually get the person to fold the other hand which is kind of like a hi. I outsmarted you. I'm smarter than you see. So if I knew that about somebody then I would know that their bluffing frequency was going to be higher than the average person's and obviously then I can use that in order to be able to win. So I was always kind of thinking about that and then the other thing is that I wasn't trying to hide what I was doing very much because I understood that because of confirmation bias because of motivated reasoning it was very unlikely that there was going to be a lot of updating going on and so I could just sort of play my game without worrying about them sort of changing their mind very much and so those are the ways that you end up like really using this and then you also understand like it was basically any bias that you could think of instead of expressing itself at the poker table. So there's definitely like gambler's fallacy which is you think of something lucky happened to you in the past it's like it'll happen again in the future. There's status quo bias like we just do things that we've always done before. I mean literally like there's endowment hands get endowed to people which is a really good one so you know that once somebody has a hand they will think it's higher value than if they didn't have the hand which means that you could get them to put more money in the pot with it. Like literally name a bias you're going to see it like really a big time at a poker table. One thing that's not publicly known is that when I engaged in the only thing that was like a high-stakes poker game in my life I turned to you for advice. That was the process of selling my company and when you sell a company for let's say millions of dollars I won't be specific. You start to have a lot of conversations where you feel like anything that you do there's millions of dollars at stake. You might raise the price by a couple million dollars if you bluff in the right way or you might lower it if you misplay your hand and I got kind of nervous doing that process because on the other side of the table were professional negotiators and I did not have that experience and so I remember calling you up was around the time you started the book and so we had reopened conversation and one of the pieces of advice you gave you was to sort of how to cope when you're not just sort of thinking in the abstract about your moves but your moves have like extra zeros. Remember that? I do actually remember that and it's actually a really hard problem in poker so it's about defining what the rules of the game are. So I think I told you to think about it as a game where your goal is to kind of accumulate the most chips not the most money. That's right part of it was that you tried to help me dissociate the money which felt a little stressful even though not someone cares about that much about money but in the moment of this kind of transaction that's sort of all that it's about and so you naturally become obsessed about the money side of things and you helped me to sort of dissociate emotionally from like play the best hand that you can from what are the numbers there. I assume that must also be part of what you're doing when you're playing for something as high as the world championship of poker that you have to like think about this as I'm playing the best game I can and not jeez if I win it's $4 million. Yeah so this is actually something that I talk about in the book is that one of the things that you really think about is what are the rules of the game that I'm playing and you can kind of define that for yourself this is a mindset situation so I can view it as what I'm betting is money and that the goal of the game is to actually win money or I can view it that the goal of the game is actually to come up with the best strategy possible and one would assume that the result of that would be money but these things that I'm using these chips aren't actually money they're just tools in order to execute the best strategy possible. Which is what Carol was doing right he was sort of dissociating himself from winning this game but what is the best thing I can do in this position. Right exactly so and I think about that actually what I'm talking in the book in terms of what what is thinking in bets do for you it kind of actually gets you into the space where you sort of redefine what the rules of the game for yourself is so I think that we're kind of we come into this world with the rules of the game or that I kind of want to end up with a really positive narrative about myself and the way I'm going to get there is sort of moment by moment in any moment I want to try to maximize how much I feel good and minimize how much I feel bad so what comes from that is that we kind of come in thinking that the goal is like I want to be right I want to think that you know the things that I believe are generally correct good things are generally to my credit bad things are generally not other people aren't smarter than I am I'm special in some way all of these things and that that drive generally gets fed by these kind of in the moment decisions so that's sort of how I think about like self-serving biases as generating is that in the moment if I want to sort of maximize the way that I feel then when something bad happens well obviously that was due to bad luck because otherwise I have to think who I made a bad decision and that doesn't feel good in the moment and if something good happens okay I'm going to take credit for that if something good happens to you I don't like that comparison so I'm going to be like oh Gary's just really lucky and if something bad happens to you I'm going to be like hi you deserved it and that that maybe helps my narrative along so we have all of these ways that we kind of come in with this idea that we want to help our narrative along but we want to do it in the moment so just like at the poker table where your natural tendency is to think this this game is about money and I need to actually have money come my way as opposed to this game is about having sort of the best strategic choices in the long run you want to think about changing the rules for yourself from I want to be right to I want to be accurate and what's the difference between those two things well if you think in bats it forces you into this place of thinking much more about do I have an accurate representation of the objective truth as opposed to are my prior beliefs true which is a completely different way to look at the world because if we think about betting the person who's going to win if we're betting between ourselves is the one who has the more accurate representation of the objective truth the one was a better mental model of whatever it is that we're betting about then the one who's just going around saying like I'm right I'm right I'm right right so we can think about this like what if you would if you express some sort of opinion that you express with like total certitude like for example let's say that I expressed an opinion like I think the Democrats based on what just happened in Pennsylvania going to take the house in 2018 and obviously when I say that there's no modulation to it if Gary then says to me well do you want to bet on what is that you do you want to bet on that what does that do that that pulls me back where now the focus is on accuracy right because I'm not going to win the bet if I'm not accurate now what I start thinking is well really how sure am I of that belief like am I a hundred percent that the Democrats are going to win in 2018 in terms of the house like am I 80 percent am I 70 percent well let me start seeking out opinions so that I can ask people who are in the right category what they think let me go look at what the polls are saying how I'll just I'm just going to start googling a lot of stuff so that I can start to create a more accurate representation of that belief so what the frame of thinking in bets does is it actually puts you into this mindset of my beliefs are under construction they're always in progress so there's kind of no right and wrong there's just trying to get closer to accurate and my predictions obviously because there's luck involved in any prediction also could never be a hundred percent so it creates this kind of cognitive flexibility I think that forces you to be open-minded because otherwise Gary's going to take all of my money in the long run and you're just going to go broke at the poker table so what happens is you have you know it's sort of reading people out right the top players end up really with this focus on accuracy because they've changed the rules of the game for themselves because they separated themselves from the money and they're just thinking about sort of what's what's the most accurate way to execute on my beliefs and I have to be flexible of me changing my mind a lot so looking at the clock yes during the rumor was that we would kind of reverse the usual roles here and swap from you wanted to ask me some questions about AI so if you want to do that now be the time and then we're going to take questions so one of the things that I think about is this idea that well maybe computers are going to solve for this right so I was very skeptical for a long time because what computers were really good at where it seemed to be like problems like chess and there's a really big difference between chess and poker because chess doesn't really have the luck element the poker has and chess also has complete information so it becomes like a calculation problem that if you have a lot of computing power you should be able to solve to the end of the game of chess but poker is like different because there's hidden information there's cards that are faced down and obviously there's quite a bit of luck in the deal and computers seem to have a really hard time with poker and then all of a sudden the AI seem to be getting better with that and then this crazy thing happened without the go and I started getting much much less skeptical and much more worried about the robot apocalypse and I was like oh no as long as they couldn't solve for these things I thought I was safe but maybe I'm not so safe now and then I read actually some really wonderful work by Gary which I want to ask you about because Gary is actually much more skeptical about AI and in this world we're not immediately worried about the robot apocalypse right so in this world where we're hearing about self driving cars and alpha go and all of this stuff happening it was a very interesting voice that I thought brought up a lot of really good points so I was I was hoping that you might talk a little bit about why you didn't don't think the terminator is going to come and kill me yeah right so the reality is to come back to the poker part in saying but the reality is that right now robots are struggling to open doorknobs there was actually a video of a robot opening a doorknob which is like major progress in the field but the video was probably staged in a very limited conditions the general problem with robotics and AI right now is that you can train something train a system to do a very narrow task go as a good example of this but even if you train them to do that it can't necessarily do anything else and they may not even be able to do a slightly different version of that so you can take alpha go it's trained on millions of games playing itself if you put it on the board that was bigger so a 25 by 25 board instead of a 19 by 19 board it wouldn't even really understand the concept of a bombing across the board so the systems that we have now are learning very narrow things another example is who is the system that learned to play Atari games also by define which did that go stuff and they train the system and it appeared to learn how to play break out better than humans this is the old game where there's bricks at the topping have to break through the wall and there's demos on the web saying and look now the machine that knows what a wall is and knows how to bounce through it so a friend of mine who I saw today at a conference at NYU did delete George from a company called vicarious say well what would happen if we just move the paddle and it turns out if you move the paddle up a few pixels the system doesn't really work anymore so it's memorized what to do when there's a paddle on you know the third row from the bottom in the seventh pixel over from the right and so forth but now if you make the paddle five rows up it doesn't really know what's going on so the conceptual understanding is not very deep the technical term for the technique people mostly use now is called deep learning but deep learning isn't conceptually deep it doesn't mean that it's induced the notion of collision and crashing through things and so forth it just means there's a bunch of layers in something called a neural networks it's a great piece of rhetoric speaking of cognitive biases like renaming the estate tax to the death tax completely changes how you feel about it so renaming neural networks which sound boring to deep learning you know very exciting people they think well you know we're on the verge of building what people call artificial general intelligence but you take these systems out of the narrow domains in which they're trained they're really not very good another example is okay now we actually have a system that can play poker pretty well but they've only actually tested as far as I know on two-player games the two-player games you can tell me much more about this than I know but are kind of different from three and four player games because we have three and four player games you have the possibility of alliances and things like that so even in the version of poker that they're playing they're taking out one of the major human components which is making inferences about what other human beings do so if you take that out you can go everybody knows you're just trying to capture the territory on the board but if you're playing poker or certainly some other games like risk or diplomacy then you have to understand what people are doing and machines are nowhere near to being able to reliably understand I did this in order to get you to do this because you thought that I would do this kind of I was thinking of Maxwell smart I know that you know that I know that you know computers can't entertain that metal level of discourse so they might be able to someday I'm not saying we're safe forever like forever is a mighty long time and if you think about like in 1994 nobody saw Facebook coming or the internet coming and just a few years ago another example today what I can't come up with it but you know people often miss major trends and you know maybe someday we will figure out how to make machines that have kind of human level intelligence and their ability to understand natural language or reason about common sense and what objects can do with things like that but it's not on the horizon now nobody has the techniques right now so I want to actually think about it I was thinking about it in relationship to this book so one of the things that I talk about in this book is that the uncertainty that's in the system right that so in chess there's very very little uncertainty right there the luck element is gone that's a really big source of uncertainty and obviously you can see all the pieces so that's also would be a really big source of uncertainty if we if suddenly you could roll the dice they landed 11 your bullshit came off the board and also maybe you couldn't see a quarter of the board so you had to guess at the position by the way that the other pieces were moving then you could create sort of an uncertain version of chess but that I'm certain that uncertainty really kind of mucks up the system and the way that I think about it is not so much because you know it's obviously it's a noisy system but because we're very cognitively flexible so I kind of think about this sort of double-edged sword around cognitive flexibility that some of the good things that you're talking about that we can do which would be like just as an example Gary talks about the fact that like in order to get a computer to understand say what a cat is it's like super data hungry I don't know how many pictures of cats who have to feed thousands of thousands of lives whereas I can show you a couple of cats sort of describe what it is and you're going to be really good at figuring out what cats are so that's this ability to generalize this like you're very cognitively flexible that that's a good thing but also a bad thing so when I think about like human biases and how we're sort of thinking through these uncertain systems that our cognitive flexibility can really get us in trouble but sort of what I hear from you is that ah but there's another side to it which is it's kind of what makes us better than these deep learning machines. I mean they're better at us at some things so they can play go better than we can. With no uncertainty. With no uncertainty. Almost no uncertainty. Game with no uncertainty and there's no there's no doubt about it that they can crush me and go anyway but take your cat example with a single trial right now I will get you to think of a concept you probably haven't thought of before which is a blue cat that's bigger than an SUV right one of these neural networks would need to see 10,000 examples of it but I can give you this idea and probably never thought about blue cats that are bigger than SUVs until just now but you have no trouble now making inferences so I can say like would you be frightened of a blue cat that was bigger than an SUV? Most people would agree yes they're not quite sure to take or could a blue cat that's bigger than an SUV jump all the way to the moon you can think about it say probably pretty unlikely I mean it could do some things that are the reason about this we don't have any clue really how to make machines do that level of reasoning and the other part of your question is really like are the advantages and disadvantages like neatly aligned so here's a case where I don't think they are even though lots of people will try to persuade me otherwise our memories are actually pretty lousy compared objectively to what machines could do so you can upload all of wikipedia into your watch or something like that now and if you know if I cared to make a watch app on the Apple watch I could and I only need to upload it once and we could make millions of copies of it and we essentially never make errors whereas you can tell me a story and I'll probably start you know messing up the details within an hour or within a few minutes so I would say objectively speaking our memories are not that good I have a theory about why which I'll share in a second there are people that have said well but that just gives us creativity and so it's great that our memories are bad compared to computers I don't think that's true you know computers can actually do creative things or quote creative things just like humans do quote creative things following certain rules and using the memories that they have that are better and sometimes you pit humans and machines and then there's not that much difference there so it's a nice story to tell ourselves that our memories are lousy because it's giving us this other advantage but the true story is probably actually closer to something Stephen Jay Gould used to write about with the pandas thumb which is sometimes evolution is moving so to speak in a particular direction it's very hard to change the direction you want to be technical about it that's about probabilities and genes and so forth but we inherited from other vertebrates and primates and so forth a memory that's really good at statistical approximation but not good at remembering specific information that didn't matters that much in the environment of the adaptation of our ancestors we got kind of all in on this particular kind of memory that's called context-dependent memory and it's very hard to just kind of rewire the brain from scratch it's not like you could take humans off line for a year and then next year release humans 2.0 that would have better memories and kind of stuck with the black narrative maybe all right let's take some questions from the audience so i actually just just one second so i just want to just bring up one point from what you said which is i think that what what you bring up about this memory problem and how we actually do this kind of statistical work is actually really interesting so i figured out pretty quickly that i was in a really interesting lab to study human cognition when i was sitting at a table with people who have been playing for like 40 years because when i started off i was pretty young i was in my 20s and the people that i were playing with tended to be people who were retirees who had time during the day to go play and i discovered pretty quickly that with a tiny bit of coaching from my brother i was actually much better than them despite the fact that they had had lots of experience now i learned in psych one i think just like every other psychology student does that learning occurs when there's lots and lots of feedback tied closely in time to decisions and actions so i thought well this is really weird because in poker there's lots and lots and lots of feedback tied closely in time to decisions and actions an average hand of poker takes about two minutes and there's lots and lots of decisions in there and then there's like a real-world consequence that you take my chips so it was a little confusing to me that people have been playing for 40 years did not seem to be that good at the game why wasn't everybody achieving expertise in poker and that's when i really started to think about these issues what i realized is that it's a little bit of a memory problem and a little bit of the way that we encode information that as these outcomes come in we encode them sequentially we aren't particularly good aggregators so we're not waiting around till i have like a thousand trials and then making a decision about all the stuff that's happened at the table instead an outcome comes in i encode it like not my fault i'm so great whatever and then that kind of gets shunted off and then however i've encoded that outcome that comes in now actually affects the way that i've i encode the next outcome because i now i have a belief about what kinds of decisions are good and bad and if the next outcome doesn't conform with that belief that i've formed i i put that off to like it's an outlier or there was something unusual or it was locked or it was part of this uncertainty category and it confer and if it confirms the belief that i had before then i reinforced the belief that i had so we have this problem which is even though this is a very feedback rich environment we take everything in sequence we don't actually aggregate it and it's all done in context to the way that we've encoded everything in the past which i think is similar to sort of the way that you were thinking about memory so i just wanted to bring that up before we take the question i'll just say that i would bet that the difference between professionals and people that play a lot that aren't professionals as professionals can probably remember their hands and what happened they probably analyze them more deeply whereas the average person sitting in a casino probably plays a hand they either won or they lost and they keep track of how many chips they have but they're not really in detail analyzing why as you said they're they're attributing you know to locker whatever if if they lost and they're good play if they want but they're not tracking it saying you know this is what i did wrong here they're not actually implying that feedback so by the end of the night they don't remember the individual hands so they can't do the aggregation right right question um i just had a quick you mentioned it but i didn't quite get it what is hand endowment oh so uh so it's not specifically hand endowment so there's something called the endowment effect which is that when we own something we think it's much more valuable than if we don't so there's a pretty classic experiment uh where um let's say that there's a visor and a mug that's sitting in front of us and people have come in and they sort of said what they're willing to pay for it so we have some objective measure from a large group of people and everybody says that they would you know the visors like you know they pay two seventy five for the visor and two seventy five for the mug so now you take a new group of people they come in and some people get mugs and some people get visors so they just randomly give Gary a visor they randomly give me a mug and now what they say is um how much are you willing to buy Gary's visor for and i saved what i thought when it was just sitting in the middle of the table well i'll pay two seventy five for it but if you say how much am i willing to sell my mug for now i'm like ten dollars and vice versa so what happens is that when things belong to us we find them very valuable and this can be true of things that aren't mugs it can be true of ideas as well so ideas that we thought of or beliefs that we have we consider in doubt to us it can be true of material goods like clothing a hand of poker if it's my hand it's in doubt to me i consider it more valuable than if it were in somebody else's possession for example so to bear is worth more if it's my to bear then it benefits your to pair exactly so you can see how this can really distort uh decision-making this endowment effect so we've talked about things being in like in doubt to somebody in doubt to me we have a hand over here let me bring him up like oh hi what are your thoughts on tells during poker game and from a psychology lesson so i love that question thank you so just for people who don't know the lingo um a tell is something physically that someone might uh do with their body that would signal the strength or weakness of the person's hand so uh we know that there's all sorts of things that people do when they're uncomfortable we know this from psychology so for example there are what are called self comforting um actions when people are uncomfortable and one of the ways that you get uncomfortable is that you if you're lying you would be uncomfortable if you're bluffing you might be uncomfortable if you feel like your hand isn't very good in poker um and self comforting behaviors would go and like if you're have you ever sort of been sitting and been doing this or you're sort of rubbing the top of your leg or those kinds of things those are sucking your thumb would be the ultimate self-comforting behavior um uh people will purse their lips or they're like do a little grimace or they'll have a uh the blink faster or they'll do something called pudding which is a very slow blink so i have to interrupt for a second um trivia fact is that um annie was uh on celebrity apprentice and like we came in second to the general rivers um so i've been you know resisting all night asking about our president but does he have any tales i think he's pretty much just all out there so i don't i don't know that the tells us tell you anything new um so so actually that's the point so there's a base rate problem yeah there are um restriction of range as we call it yes that's right so okay so so the thing about tells her that like they're super sexy right like if you can say like oh i knew to full because um you know he blinked his eyes that's an awesome thing to be able to do i actually think about tells us something very dangerous because in general most most poker decisions that you're making aren't actually very close um and we tend to have much more confidence in uh our ability to sort of get to a certain place than we actually should have so let me give you a very simple example that's very mathematical let's say that i'm gary bet such that um there's uh nine hundred dollars in the pot and i have to call ten dollars to win that hundred okay so what you can imagine is that obviously when i win the pot i'm gonna get a hundred dollars and when i lose i'm only gonna lose ten dollars so i have to be right a very small amount of the time in order to actually make that be correct for me to call um so like for example if i'm right ten percent of the time i'm making money okay now i'm of the opinion that i've never been ninety percent sure of like almost anything in my whole life so i actually don't want to think too hard about that call as long as i think that there's like some chance that i can win meaning it's just us and like i have a high card in my hand i should probably be calling there and i don't really want to talk myself out of the call very much and i don't want to be confident in any greed that i have a gary i want to sort of actually ignore it according to myself off from it because statistically it's just i just i can be wrong so often there and make money so you bring poker towels in when you're not sure when it's really close you're really like i'm so completely on the fence here i don't know um so that's number one because most of the time the math is pretty wide so you actually wouldn't want to talk yourself out of whatever you think the obvious choices and tells can actually be a way to convince yourself to go away that would actually be mathematically bad so it's really when you're sitting on the fence number one and number two it's also really important to realize that you can only read a tell right at the beginning because what happens is if i'm staring at gary for a long time he gets uncomfortable like do you like to be stared at so if i'm thinking about a hand very long and so now i'm like you know i've been staring at gary and it's 45 seconds in and now i decide i'm going to take a read on whether he's uncomfortable to try to decide whether i should call her full i'm actually gonna that's going to be really bad data because he could be uncomfortable for all sorts of other reason so what i try to tell people is it's actually much better to just try to think about how often does this person generally enter a pot how often do they bluff you know how often do they have it what have they what have i seen what kind of cards have i seen them turn over in this kind of situation in the past and that kind of information is actually going to be much more high fidelity do you think that phil helman is is better than average at this and he's supposed to be famous for reading tells right actually i so phil is i mean he's a great player and he's definitely really good at reading tells the thing that phil is really good at and by the way i just want to say i really like phil i think he's great he gets people really aggravated at him and i cannot tell you the value of having people aggravated at the table like because they just like they get because if somebody's aggravated when they're playing against you it means that their limbic system is on fire and when someone's limbic system is on fire which is the emotional control center of your brain just so that you guys know that that has an inhibitory effect on the prefrontal cortex which is where your actual rational thought might occur and so what happens is when you get people aggravated you actually make it so they can't think straight and he's really really good at getting people you know getting other people's skin he's also very good at like reading other players as well but i would say like his main superpower if i had to think like what is phil helman's superpower it's like getting other people's skin when they should play it with the oh hold on i have this question right here all right getting back to the um oh god i can't think of the thing with the mug in the visor what did you call that endowment effect okay all right let's say this scenario is um you said when you own something it's more valuable let's say you own the mug and you're a big coffee trigger gary has the visor be preferred over sunglasses i say if somebody wanted the visor he would sell it for 50 cents because has no use to him now i have a scenario where instead of a physical object this is the scenario i'm a math teacher but i'm also math i call myself a math professional math petition because i'm published and i i'm on the speaker survey in fact i came from red versus morning and i demonstrated this whole thing it's some all right and a good friend of mine is an attorney let's say now here's what happens if anybody can say no i'm mathematician but we say could you explain to me what the golden rectangle is hey i'll do more than that i will construct it for you i'll show i the art teacher my school asked me how to do this i will show you would you like to know anything about prime numbers blah blah blah okay so my friend uh listen can you can you help me with um can you just tell me something about the worst situation but that'll be 250 an hour why is that why am i willing to keep it all the way for free and he won't free the word of of anything illegal without any paper why would that be why don't we both take a guess at this so i i think it's the frame from which somebody comes from so a lawyer is set up that they charge hour leave right so their their advice is considered to be of some value that there's some now market value to that people have agreed upon when i talked to you you have to pay me 250 an hour and he takes that frame outside of that i get paid isn't he right but go ahead he's in lieu of making the lawyer jokes that are screened to mind we only have time for about one or two more questions yes i'll hold it back question back here and then maybe we'll take one all the way in the front thank you uh the question i was wondering through your poker experience if your risk aversion uh the in terms of the utility function if your as the reward increase did you become more risk covers as the one increase i don't know what oh and the reward increase right so this actually um thank you for that question because that relates back to what gary said so one of the big problems that happens um with poker players one of the one of the things that kind of will will cause you to plateau is that you start to think about what the stakes are that you're playing for so that's how the reward gets bigger right in general is that you're playing for higher stakes which means that you're thinking about it is money that you're moving around as opposed to uh these chips or tools much like a go pieces and i just need to move it to the right spot on the board in order to try to win so being able to create that separation is actually really important or that will happen to you so in general if you took the average person as you increase the stakes they become much more risk averse because generally you're trying to you're essentially trying to minimize regret right so you're protecting yourself against the downside obviously which means that you become more risk averse right so your what you're actually trying to do is make sure that you're viewing your chips not as money now on the flip side of that when you know that someone's acting as if there's some sort of scarcity you could take advantage of that so if i know that gary is worried about the stakes that he's playing at or he views his chips as very scarce like he's in a tournament situation i can actually push you around more and get you to to fold with too high of a frequency i can get him out of the pot too often because of that issue because he's i know that he's going to be risk averse so one of the skills that you develop as you move up in states is being able to sort of divorce yourself from it and not not think about the reward um as money but just this sort of chips no matter whether and what's reverse of that by the way is that sometimes i would go and play small states with a friend of mine like much smaller than i would normally play a table where i might buy him for a hundred dollars when i'm just coming from a place where i might buy in for a hundred thousand dollars and i would play exactly the same as a matter of discipline because the reverse can happen if you really jump down in states you can become too reseaking and i always wanted to have the discipline to just view it as a game no matter what to paraphrase a line from the wizard of us pay no attention to the number of zeros behind the curtain yes all right one more question thank you uh from many things you said i i believe that much could be learned by economists from the game of bridge and i don't know if that i really don't know if there's penny a point bridge you know bridge for money uh i know the tournaments and someone don't know what the prizes are but um i wonder if uh a congress scientist has ever done what you have done with boker with bridge to um i think even i think frankly that what could be learned i don't think there's actually been that much research either in artificial intelligence or in psychology on bridge it might be because it's a four player game and you have it's a little bit more logistically complicated so one of the reasons that go has succumbed is machine could just play itself and you don't have any of these kind of theory of mind questions about am i making a deal is there um you know trade with the other person and so forth that do come up in bridge i think you're right that it could be very interesting and maybe that will actually be one of the next frontiers in game playing i would put an asterisk on that which is i think there's too much emphasis in a i right now on gameplay there are lots of other interesting questions out there but if if i were going to take a next game that that might well be it so it turns out i believe that we have some books that you are are able to sign are they back there is that how are we going to do this we'll be down books finding downstairs refreshments upstairs so let's give any come get a book and have any sign and thank you very much