 I'm ready to turn the mic over to tonight's introducer and he'll let you know why he's a very appropriate introducer for tonight's speaker for a very special reason but I will just also tell you that it took tonight's speaker and me several breakfasts to convince our introducer mathematician Ken Ribbit that he needed to skip his last class of the semester and come here to introduce the speaker and I'm delighted that we were at last successful after a couple of no I couldn't possibly do that and so we're very grateful that Ken Ribbit is here with us to introduce tonight's speaker. Thank you very much Cindy very often when I come to introduce the speaker I have to rely on notes and actually tonight is a little bit different because the speaker this evening is actually my wife Lisa Goldberg and so I know quite a lot about the speaker and can tell you about her with great confidence on the other hand I make no claim whatsoever to being a disinterested observer before introducing Lisa as a person as an individual I just want to say that the two of us grew up in the New York metropolitan area I was born in Brooklyn before there were hipsters there and we both went to public schools I went to Farrock Way High School which no longer exists Lisa was a professor at the City University of New York when I first met her she worked at the Graduate Center which was on 42nd Street at the time and in Brooklyn College which many of you know we are the parents of two wonderful daughters who are now independent and in their 20s and now I can talk to you about Lisa Lisa was trained in a subject in mathematics called algebraic topology when she came to New York to work with Dennis Sullivan and John Milner and other mathematicians whose names are familiar to the professionals in the audience she began working in a subject called complex dynamics which moved her quite a bit from her original focus in mathematics and then when she moved out to Berkeley she moved even further she became one of the pioneers among the mathematicians who began working on Wall Street and elsewhere in mathematical finance and I think Lisa's move exemplifies the fact that mathematicians who are trained to think very carefully and precisely and deeply about an individual subject can be very diverse and very mobile and work in different subjects this is a phenomenon that our colleague Mark Green from UCLA calls the mathematical diaspora and Lisa is now very well known and very accomplished in the field of mathematical finance she has a string of patents to her name also many articles in academic journals and many articles and practitioner journals in journals for mathematical finance professionals she currently works as director of research at a portfolio management company in the San Francisco Bay area called a period she's also co-founder and director of research for a organization on the UC Berkeley campus called CEDAR which is the Center for Analysis of Risk and in these two roles Lisa works on many questions involving mathematics and works with students and colleagues she's a great collaborator as many of the best mathematicians have become and tonight she's going to tell you about something that's not related directly to mathematical finance instead it's related to basketball and so Lisa I call upon you to talk about the hot hand. Thank you dear that was very nice and thank you Cindy for the kind invitation to be here tonight thank you MoMath for putting together this amazing event I've been watching the videographers and the sound people and I had no idea it was this complicated to tape a math talk so I'm quite overwhelmed I'd also like to mention the title is a little bit West Coast do Steph Curry and Clay Thompson have hot hands so maybe you guys have heard of them if not they play for the Golden State Warriors and they both score a lot that's those guys yeah they're currently playing in the second round of the final postseason playoffs against the Houston Rockets and it's been an interesting series from a number of perspectives so far like to mention my co-authors Alon Docs who's here in the front row I'm delighted to say and Nishant Desai who is 3,000 miles away but I hope I don't know I hope he's thinking about this and we'll tell you a little bit about this talk Ken said that it's not really that it's different from that mathematical finance actually there's quite a lot of overlap between sports statistics and the way financial markets work on all kinds of levels parallels through gambling parallels through economic systems involving salary caps and distribution of wealth and so I'm not the only one there seems to be all of a sudden a whole bunch of articles coming from finance professionals and going out on the web so I think we're going to be hopefully the two subjects are going to be learning more and more from each other working on this paper is not so different I find from working on financial economics certainly not as different as say algebraic topology is but as Ken said one of the great things about mathematics if you learn it is that you get used to the idea of asking questions and getting to the bottom of things as well as you can mathematicians more than any other group I know refuse to take things for granted and try to understand deeply and it's been a skill that has has been good for me all of my life I do mathematics every day I started very young and it's always fun now in this particular research project touches on many different themes that have become interesting and important to me and so maybe in life changing over time mathematics is certainly one of those and others will emerge as as we go through through this talk obviously basketball is going to come up but before I talk at all about basketball I want to talk about something that happened to me seven years ago I I had a job in industry for a long time at a company called bara and we built and bara still does build portfolio risk management tools and models and software which are all over used by by financial services all over the world so if you have a 401k planner you manage money in any kind of way you're probably touched by what that company did just and does which is quite amazing but after being there for more than for almost two decades I left taking quite a risk in the sense that I didn't make a new plan for myself it was May of 2012 and I just kind of walked out the door and all of a sudden I had lots of time and so when you're in that situation if you've ever been in a situation like that it's kind of interesting what you pick up I was had been book review editor for a journal called quantitative finance it's one of these medium journals between academic and practitioner that's kind that Ken was mentioning and so I assigned myself a book to review and that book was called is called thinking fast and slow by Daniel Kahneman so I wonder if any of you know that book yeah lots of people know that book Daniel Kahneman before I tell you about the book I'll tell you about Kahneman he is a psychologist and he won the Nobel Prize in economics in 2001 and so you can ask why is a psychologist winning a Nobel Prize in economics and it's because the work that he did in collaboration with a guy named Amos Tversky who I'll talk about quite a bit laid the foundation for behavioral economics and behavioral finance and I had always wanted to learn about that we didn't do very much of it at Barra and this was my opportunity so I got the book it was very popular an amazing thing Kahneman wrote this it was a best-selling book a huge best-selling book when he was say he wrote it when he was 77 years old so that's something to think about and this book is so so this book is a compendium in part at least the first two-thirds of it of the research that Kahneman did with Tversky Amos Tversky his partner who did not win the Nobel Prize in 2001 why is that because Tversky died in 1996 of melanoma Nobel Prizes I've always been wondered why this is but they're not given posthumously so there's Kahneman he's winning this Nobel Prize in a subject that's not a subject for work which he regarded as something that was a partnership and this this book is very much a tribute to Tversky and so what is it that makes behavioral finance and behavioral economics go if you've ever taken an economics course standing in the middle of it is Homo Economist or this guy who makes a lot of rational decisions it's got to decide whether to invest in stock A or stock B and so he kind of thinks through all of the rational implications of this choice and then does something optimal so we all know that people don't really act like that in fact people do very emotional things and what Kahneman and Tversky did that has translated into the subject of behavioral economics is understood in a systematic and deep and novel and remarkable way how it is we make mistakes deviate from rationality and I'll just give two examples because a lot of the stuff that has grown out of their work has become very familiar today maybe you guys have heard of like hindsight bias which is that you know the world looks very complicated when you look forward into the future but when you look backward you think well you know I should have known that or I could have predicted that or it makes a lot of sense that it turned out that way right well it the reason for that is because the way we remember we remember the things that turned out to matter and we forget a lot of other stuff so so this is this is a bias another one that's very important is called confirmation bias which means that when we're out trying to figure out what's true and what's false we tend to have an idea in our heads about it and we look for evidence that confirms what we believe instead of looking for evidence as we should do if we're good scientists that tears down our theories or we should be we should be balanced about it but it's very hard now why is it that we do these things many of the biases that we have are linked in fundamental ways to the way that we misunderstand randomness and this is really a theme that runs through all of Kahneman and Tversky's work seeing something it's out there maybe this is just chance events but instead we look for patterns in what we see whether there are patterns there or not we're pattern seeking individuals so if I say to you one two three you might say four five six right it's a natural progression or if you're a mathematician maybe would say five eight thirteen right you might make a Fibonacci sequence those are both patterns one's a little bit more esoteric than the other but hardly anybody would just throw out three random numbers we don't do it we don't like randomness we we look for for meaning in what we see and so that has I would say that book has changed the way I think about everything and I try to always keep in mind what I've learned from Kahneman's book and one more thing and this is perhaps the most amazing is that you we can be as introspective and as conscious as we want of the types of mistakes that Kahneman and Tversky have documented that we make and we will make them again and again and again you might be able to get out of it sometimes by really focusing on a specific thing but your mind will let go and you will make these mistakes over and over again you will tend to try to confirm what you believe you will tend to try to see patterns when there may or may not be any you will tend to look backward and say you know I should have known that all along so that is certainly kind of an important element of of the work that I'll present here so sports is another theme particularly basketball I grew up watching basketball in fact I grew up watching this team these are the New York Knicks from the 1970s when they were they were they used to dominate they used to dominate the league so this is Dick Barnett Walt Fraser the guy in the middle there without his shirt that's former senator and presidential hopeful Bill Bradley Dave de busher and Will's Reed these were certainly not all the Knicks that I'd like to talk about it was the best picture I could find though so you wouldn't what you would want to talk about Kazi Russell about Jerry Lucas certainly about Earl Monroe this is yeah or all the pearl right I mean these guys were amazing and I loved watching this team in fact my my parents were here for the afternoon session my parent my mother's in her 80s and my father's in his 90s and my brother unfortunately couldn't make it but we would sit around and watch game after game this is like a really important part of my childhood and we would always talk about who had the hot hand because it was it was very often this guy Walt Fraser he would start shooting and it would just seem like he couldn't miss and this is a very real phenomenon I watch a lot of basketball now mostly the Warriors but this can will tell you I've kind of dragged us into many many other teams as well not just the local ones we were watching all the time and when you see a top player start to score you really feel like something has changed in that player and that player is a good bet has a hot hand get that guy the ball he's gonna he's gonna score for your team so everybody knows about the hot hand players know it fans know it coaches know it God knows commentators know it here's a quote typical from the old days 1970s out of the Times Bill Bradley has taken over the hot hand from one of his teammates and here are some photographs that can took when we drove by Oracle Arena which is where the Warriors have played and are still playing for the next few games I'm sad to say they're moving into San Francisco but if you drive up and down up and down 880 in California you drive right by the Warriors and it's been like a big shrine with giant pictures of the star players and a big hot hand sign out in front of it and you might say well it could be surprising to see as late as 2017 to be seeing the hot hand or hearing about the hot hand because in 1985 which was like 10 years after I was watching the great New York Knicks but still a long time ago today three researchers just published a study saying that there was no such thing as the hot hand it was a cognitive illusion so if you look at the names I maybe should have highlighted them three here on the study Thomas Gillovich Robert Vellone and Amos Tversky that very same Tversky who did not get the Nobel Prize but was a real expert in seeing in how people see made-up patterns and stuff so these three researchers published a study arguing that there was no such thing as as the hot hand the result was surprising and while scholars were very excited by the result and really excited so if you're if you're in a scholarly discipline maybe some of you are you know you work when you write a paper you do a study or you prove a theorem or you work for like years and then you work for more years trying to write the thing while and make sure that it's something people will understand and enjoy and then you put the paper up and nobody pays the slightest bit of attention to it and this is something that I think has happened to everybody in academia even even the more successful academic academic admissions have stuff that they work very hard on nobody paid attention to but everybody paid attention to this paper all of a sudden I mean if you look up the hot hand literature that cites it there are thousands of papers like academics thought wow everybody's misunderstanding something how cool is that let's go see what else people are misunderstanding on the other hand the NBA didn't think highly of the result so this is red our buck who was a famous coach and president of the NBA and he look at the thing and he said you know this guy's got to be kidding what do I care of course there's a hot hand and so if this sounds kind of familiar like the the experts who have the experience and the knowledge and the intuition versus the egghead academicians who are too smart for their own good and think they know better than everybody even though they have no experience with the subject there's that kind of thing going on today maybe over climate science or if you've read in sports again if you've read Michael Lewis's Moneyball and so who's right and who's wrong well it's kind of a question so let's see how a little bit more about how the original study worked so in the original study these guys did something that maybe is very elementary but also kind of it's nice to do simple things they took a player's performance and represented the thing is a string of zeros and ones zero if you miss the shot and one if you make the shot and then they went to measure hot handedness in the string they went to see if there were unusual streaks or maybe after a streak if there was more likely to be another hit they've made a measurement now I'll tell you exactly how their their measurement work in fact in their in reality in their paper they did lots of different measurements that are pretty similar but they looked at the problems from from several different ways and then they used statistics to decide whether something was unusual going on in those strings that came corresponded to players shooting records compared to other strings it's pretty pretty simple pretty straightforward they found no evidence that star shooters had hot hands in fact for some of the players they found an anti-hot hand they found that the player seemed to be more likely to make a shot after he just missed some and according to the statistics they used that the difference in the wrong direction was statistically significant so this was their findings the findings that read our back thought he should not pay any attention to this is just ridiculous and there are in fact a lot of reasons a lot of reasons why they might have gotten that result one reason might be that the representation of performance was just much too simple so here here is an example this guy in the red is James Harden who was the most valuable player in the 2017 2018 season we'll see if he gets it again I don't know and here he is making a relatively easy shot where the defenders are behind him and here he's making another kind of shot where he's he's got two warriors between him and the ball so if he made those they would get a one if he missed them they'd get a zero but obviously the difficulty of the shot is different and this was not accounted for in the original study also the dynamics of the game of basketball what happens maybe maybe James Harden starts getting a hot hand and his opponents change their strategy to shut him down right so accounting for the defense and there are a lot of papers that try to improve on criticize reverse I don't know what you want to call it the original study by taking these more nuanced and empirically obviously important effects into account so that's that's worth doing if you've ever published so I don't know maybe there's some empirical researchers here this is pretty typical if you put out an empirical study I mean people will criticize every if they pay attention to it at all they will criticize every aspect of it it's not realistic you should have used a different time period this is overfitting I mean there you can criticize all of them and these are valid criticisms when you start making something more complicated though you do tend to introduce new kinds of errors that you didn't think of making something more realistic is not necessarily making it better so you can decide if you want whether you think the more simple result representation that neglects these things gives you a stronger point or a more complex representation that takes account of them makes a stronger point I think the original authors like their explanation they like the explanation that people were just misperceiving what was going on they were not streaks in the data people felt there were streaks in the data because they were very excited about the basketball and so they made up some story after the fact to explain what had happened so here's an example of looking back on some data and creating creating a narrative around it. Tversky has been lauded by by many for being incredibly insightful on the details of how we misunderstand randomness this particular quote comes out of Michael Lewis's book the undoing project so for those of you who have not heard from of Kahneman and Tversky probably everybody's heard of Michael Lewis he's one of the most famous writers in the United States right with money ball which I mentioned before and the blind side and a lot of books he wrote a book called the undoing project which is the story of Kahneman and Tversky's friendship and research partnership and also gives quite a bit of technical detail as only Michael Lewis can in a way that's very very accessible it's a super wonderful book if you haven't read it but Lewis in his two thousand and I think seventeen book just like Kahneman in his two thousand and eleven book the thinking and thinking fast and slow praised this hot hand study as a testament to how insightful a Mr. Tversky was about cognitive issues and perception so here are two competing explanations for some study but in two thousand and fifteen when I was sitting in a seminar I heard a third explanation one that I had never heard anything like before and these two guys Josh Miller and Adam Sanjerjo explained that the reason that Tversky and Gilevich and Valone had not found a hot hand is because they did their statistics wrong there was an actual mistake in the study so this is like amazing because these were kind of God like academics extremely influential and this is not saying you didn't account for something this is like you screwed up and so I can direct you to their paper for the full details goes on and on it's a paper in a journal fancy academic journal called econometrica that talks about what they found I heard this as I said in a seminar Ken mentioned we have a research institute at Cal called Cedar this consortium for data analytics and risk we have a weekly seminar and these two students were in the seminar giving a talk they were students jointly in the law school in the statistics program and they were building a robotic judge and one of them happened to mention this mistake and of course having spent so much time on conomans book and a lot learn trying to learn as much as I could about behavior since I knew all about the subject I was amazed that there could have been a mistake I certainly didn't notice it and I sort of I confess stopped listening to them a little bit and started looking I had my laptop with me as I do started looking for the error and indeed it is a mistake so I'm going to try and explain this is their explanation not mine how the thing works so it works by playing a game you can see the entire mistake in this game so we're going to flip a coin and we're going to think maybe the coins like a shot string so it should be like 20 or 25 or 18 18 flips in a row but you can already see that the issue for three times and so I'm going to just play a game where we flip a coin three times and this coin is a fair coin so it's 50 50 heads tails and the flips are completely independent what happens on the next flip has nothing to do with what happens on the previous flip and so the way the game works is that I flipped the coin and you have a piece of paper and when I flip the coin the first time you do nothing you pay attention and if it's a tails you just sit there but if it's heads you become alert and then after the heads you write down what happens so flip tails you do nothing I flip heads you become alert if I flip heads you write a second time after the heads you write down a heads okay so you're writing down what happens after I flip a heads and doing nothing else so 50 50 coin the flips are independent and the question is what do you expect to have on your piece of paper after we've played this game half heads half tails right we're writing down what happens after heads and of course what happens after heads has nothing to do with what just happened because the coins are independent so you would expect to have 50% heads and 50% tails let's switch fursky thought but it's not true and it's not true as you can see here there is a small sample bias so in a string of three these two to the three power equal eight things are all the things that can happen so if I were to if I were to flip three tails you would have written nothing if I were to have flipped tail tail head you would have written nothing now here you would have write something down tail we do nothing heads we pay attention we write down the tails here heads pay attention right away write down the tails now you do nothing for this one you get an h and a t h th you get only a t again right because you don't do anything after that last heads here you get only an h and here you get two h's so if you look at the fractions on your pieces of paper in the cases where you actually had a heads you actually had a shot you're thinking the heads is like a basketball shot and the tails is the heads is one that made and the tails is one that missed 0% were heads 0 here is a point 5 0 1 and 1 and if you add them up and divide by 6 you get something less than a half 2.5 over 6 is definitely less than a half it's 5 12 so this is kind of remarkable I think the calculation is very easy to do what is takes thought is to ask why this is relevant to the problem that we were working on and the answer is we're looking to see what is the fraction of shots you make or miss after a string this is exactly the predictor of it it's an estimator it's an estimator that's biased downward why is it biased downward it's biased downward because of the three if we had done this for an infinite string it would have been 50 50 but as long as you do a finite string there's a downward bias the downward bias which is the difference between 0.5 which is what we think we should have and the actual estimator these are the number of flips the bias gets smaller as the number of flips gets larger if you go all the way to infinity that bias disappears but it's actually pretty big around here around 20 and if you look not just at what happens after one heads but if you were looking at what happens after two heads or after three heads or after four heads were the same principle is still there you would think it should be 50 50 because it's a next shot and they're independent these biases are pretty large and these are these empirical estimators are what are called in statistics consistent estimators which means as the number of flips goes to infinity the bias will go away but it's there at every finite level so this was the insight that these two statisticians Sangeurgeau and Miller noticed after 30 years of everybody arguing about this paper so what do they think well here are some implications of the bias first of all in a finite sequence of independent coin flips when the coin is fair a reversal is actually more likely than a continuation in the original experiment if you'd accounted for this which the original experimenters did not it made it easier to find a hot hand making you think that if you redid their experiment maybe the results would reverse now as we know in mathematics if there's an error in a proof that doesn't make the conclusion wrong sometimes it is and sometimes it isn't the authors who Miller and Sangeurgeau who found this small sample bias claimed that after they redid part of the part of the original experiment they found a reversal in the results and I don't think I said it earlier but in the original paper Gilovich and Valone and Tversky looked at three classes of data they looked at the Celtics shooting field goals they look sorry the 76ers shooting field goals the Celtics shooting three free throws and then they lined up a whole bunch of Cornell students and said you take a hundred shots and it was a controlled experiment and it was in the controlled experiment that they found the reversal they have two very long footnotes in their paper explaining why they didn't check the other parts they didn't think it was relevant I don't know maybe they didn't have the data for us we thought it was the most relevant part because what's really interesting anyway to me and to Nishant was what's happening in the game but before I explain the experiment we did I just want to say for those of you who do play in the stock market the small sample bias tends to reinforce the gambler's fallacy which is you know I just my stock went down ten days in a row I'm due for a reversal well no you're not but everything we see around us tells us that reversals are more common than continuations all else equal so if you're interested in doing a psychology experiment you could actually like test on human subjects and see does this really you know subject them to subtle subtle reinforcements like that and see how it affects their behavior I don't I don't know if anyone's done that but I'd be curious what the results would be anyway it's there and and we know as we get more and more deeply connected to the internet which we seem to do every day there's all kinds of stuff being pushed at us that suddenly affects how we think so so let me explain how the statistic works so this is a particular shot string for Clay Thompson Clay Thompson is my personal favorite Golden State warrior he's very calm and kind of streaky shooting wise compared to others but as a defender he's like reliable and solid and non-emotional and very interesting player this is the performance that he had in a game against the Detroit Pistons back in December of 2016 right before Christmas and this is 16 shots long and he made half of them and he missed half of them so I'm gonna explain now the kind of central statistic that was in the original hot hand study what they did was they looked for all sequences in a row of hits in this case we're gonna just look at instances of two hits in a row and then we see there's two of them here there's one of them here and there's one of them here so there's four instances of them and we can look to see what happens after these hits well we got a one we got a zero we got a zero and then we don't know because he took 16 shots and the HH came at the end so empirically in this string Thompson the probability the empirical probability of a hit given two hits was a third Tversky and and Gilovich and Valone compared that to the probability of a hit given two misses so here we've got five instances of two misses none of them at the end so I can work with all of them and what we find is that after the first two misses there's a hit then three misses then a hit so the probability of a hit given two misses was two-fifths the kind of instinctive thing is that those things should sort of cancel each other out and so this minus one-fifteenth looks like the antithesis of a cold hand but because of the small sample bias we might expect the statistic which we've named T2 to be a little bit less than zero for for random strings so maybe maybe this is something we should look at the likelihood of a hit after two misses was higher than it was after two hits this sort of seems like the anti-hot hand but because of that small sample bias we need to be careful so now I'll come to the next theme I've talked about mathematics we've talked about behavioral economics talked about basketball and mathematically now I'm gonna talk about math teams because as a much younger mathematician I somehow learned to do to think that I had to do everything myself and when I gave a talk or wrote a paper whether it was with collaborators or not and there were not that many collaborative papers back in the old days I felt like every detail of it was my responsibility and this is kind of a great instinct in a way but the flip side of that is that in some ways you're limiting what you can do if you insist on doing only the things that you understand perfectly and having a bigger team of where your collaborators have skills and knowledge and and experience that you don't let's you work on bigger problems so I had this idea that I'd love to be able to see if this bias effect how did this all play out in modern data and I had absolutely no ability to do that because I'm not good at data science or web scraping I don't really know how to program I'm not good at designing algorithms all of these things and my colleague Laurent L. Gowee introduced me to Alain and to Nishant who had all of basketball at their fingertips and were great at all of these things so they were not only very mathematical but also had many many skills that I did not have and we figured out a way to look at whether modern basketball players had hot hands or not in the spirit of the original the question was did the original experiment applied to today's players what conclusion would you get accounting for what we know now about the small sample bias so here's Steph Curry who is generally everybody's favorite player and there's clay again and we looked at the Warriors 2016 2017 season there were 82 games in the regular season and there were I think somewhat to the Warriors chagrin 17 games in the playoffs now they won the NBA championship in 2017 and they would have liked to have win it in a straight sweep there's four rounds four out of seven series and no basketball team today still has ever swept through four four four and four and they almost did it but the Cavaliers that had LeBron James at the time beat them once so there you go we end up with 99 games there were low injuries that year Curry played 96 of them Thompson played 95 of them and what we did was we calculated this t2 statistic for each of the players in each of the games now here this is kind of a choice another empirical choice if you watch basketball you might think well the player gets hot for a quarter and then cools off for something this is our perception so why did we not use quarters well even the best shooters don't attempt more than 20 25 shots in a game so you would have not enough data this is a compromise it's not really the right length of time to address the thing we're trying to get at but it's what we had and even this size data suffers from what statisticians called low power we wish we had more data we don't really have it what we did do was imagine that the team without paying attention to who is shooting but the team as a as a whole might have gotten a hot hand and certainly does does happen sometime team the Warriors during this season the average shots that the whole team took in a quarter was not so very different from what star shooters would take over the whole game so we looked at that as well these shooting percentages are a little bit higher than you may be used to if you watch basketball it's because we included both free throws and field goals two points and three points in the shot strings we didn't distinguish if we distinguish we would have lowered the power of our study further we did rerun the study without free throws and we didn't get a different answer than the one we got but these are just choices so here's our paper you can find it in the mathematical intelligentsia and also reprinted in scientific American from last October just as the NBA season launched and what we did was we took each player or the whole team the each player for game or the whole team for a quarter and we looked at the shot string through the lens of a permutation test so we computed that t2 statistic where I got minus 115 before for one of clay's game and then we took the string and we scrambled it there by wrecking any relationship between what happened before and what happened after and computed t2 again and then we did permuted it again and we did t2 again we can just do it on all the strings or on a very large sampling of them and this sort of test which is so easy to do now that we have a lot of computing power was not it's not really not been available for all that long and it certainly was not something you could have easily done in 1985 does not suffer from the small sample bias because we are including the size of the string in figuring out what's ordinary we didn't assume anything of it what happened for long strings in trying to assess the statistic on a short string all of our comparisons are on the same size so the bias is just corrected by itself there's kind of an interesting point about this which is at Berkeley now there's a new data science division and if you go into that instead of into traditional statistics you will be taught statistics using a lot of these finite sample permutation and bootstrapping type methods and students now think that's the correct way to do it and of course you're trading in one set of assumptions for another there are cases where you would be much better off using analytical formulas and some type of asymptotic assumptions but it I sometimes fear we've we've just we haven't really educated critical thinking as much as we could and just made a kind of a trade but in this this particular case if you'd use this methodology you would have just gotten the right answer instead of having the mistake so what we do is we create T2 for all these strings we look at the fraction of T2 values that were bigger than what we observed we get something called a p-value and if you got a small p-value that means that the guy had the hot hand in that game or the team had the hot hand in that quarter and so going back to Clay's string that we analyzed here is what you would need for the hot hand and here is where Clay fell even though it was negative it was kind of right in the middle but it was nothing extraordinary so according to this metric which you can decide whether you like or what you don't like but it's the one that was emphasized in this famous study no hot hand for Thompson in that game even with the correction so when you run experiments lots of experiments you may have heard terms like p-hacking or multiple tests hypothesis testing if you if you look long enough something unusual is gonna happen so you might think that if you take a player like Steph Curry even if he doesn't have a hot hand he's gonna have a hot game once in a while so you might expect to have a small p-value say one that is less than the sacred 5% that is dominates all of empirical study that's medical studies and social sciences this is just a standard cutoff that people use for reasons that I don't really know and don't know anybody who knows but it's certainly the the practice you would just expect to see that 5% of the time generalized and said a different way if I took the p values over the whole season and took cumulative sums if there were no effect I'd see a perfect 45 degree line or something pretty close to it so this is what we got some pretty good-looking 45 degree lines they're not absolutely perfect but given here the hot hands down here I mean you just don't you could argue about whether there's a little effect or a big effect a little effect or not but there's certainly not a big effect there are lots and lots of games where there was just no no hot hand these things were pretty uniformly distributed on the zero on the p-values so we found that even after correcting correcting for the small sample bias that the original hypothesis seemed to hold at least in this study now going back to the division between the kind of geeks the statistician people and the the very experienced and knowledgeable intuitive practitioners well it seems now we've come kind of a little bit reversed in a way well not quite reversed we've got the disagreement in a different place ever since the finding of Santorjo and Miller now we've got they have there's been a new stream of literature making very detailed analyses with lots of bells and whistles finding hot hands all over the place in spite of our study so we no longer have the statisticians versus the practitioners we have some of the statisticians versus the other statisticians our study has been I think literally criticized for being mathematically correct but empirically wrong so okay I'll take it this is one of the most hot hand performances that I've ever seen it was a little bit before the game I was showing you before it was clay Thompson again he did something truly exceptional by scoring 60 points in a game if you don't follow basketball like 120 points is a pretty high-scoring game so I don't remember what the score was for the game but he probably scored roughly half the points or maybe more than half the points by himself that's certainly rare he shot the ball 44 times he made 31 of the shots but did he have a hot hand according to this statistic well no here is like the hot hand region and he was nowhere close so if this gives you like pause to say is this the right statistic that's maybe a very good question there's a lot of measuring that we do if you read studies you'll see you know the word like or like the GDP or you know how big is unemployment or do you know how many people have this disease we can never measure these things so scientists social scientists physical science well physicists are a little different they actually can measure things sometimes but apart from them we can't measure things precisely the things we want to measure so we make up these statistics and then we draw some conclusion about them and then we apply that conclusion back to the thing we wanted to measure often without maybe thinking as carefully as we need to about whether the thing we measured has all that much to do with the thing we were trying to measure so according to this statistic clay didn't have a hot hand in a 60-point game which I think everybody would disagree with it's as much a statement about the thing we measured as it is about Thompson it's certainly unusual to ever score 60 points in a game and it's unusual to score 70% of your shots so especially when you took that many of them so there was something very unusual about that game but it clearly did not appear in this measure of the streakiness of of the sequence so so that is that that is most of of what I have to say just to summarize that we picked an experiment deliberately to mimic something that was done historically to see how a historical conclusion would have fared today even after there was an error in it seems to have fared just fine although we may have learned a little bit about that statistic especially looking at those funny histograms might make you think a little bit about them too we did find that there was no evidence of hot hand in the original sense of the measurement by these three very famous guys and it was true even though correcting the small sample bias lowered the bar for finding exceptional behavior but this is not all there is to the story there's arguments going on about the hot hand all the time in newspapers in academic journals and sports do winners repeat certainly on Wall Street right as this managers luck or skill as a success we hear these things all day long and the one thing that I am very sure about is that thing I learned in 2011 when I read Kahneman's book is that the way in which we misperceive random effects look for narratives in them insist that there's some reason why this thing happened or that thing happened is very profound and seemingly uncorrectable so that's Tversky on your left as you face and Kahneman on the right by 1985 when Tversky's paper was published both of them were already famous but they weren't born famous the paper that put them on the map is called belief in the law of small numbers so if you know the law of large numbers this is a joke right the law of large numbers is that if you take an average over a whole lot of numbers it's close to the true average if your numbers were drawn independently out of a out of a sample right so large data sets can tell you things but what they were this was one of the first error types that they documented in literature in 1971 in this paper that we tend to draw too much inference from small samples it's a human tendency it's maybe keeps us get eaten from bears right you kind of take eye kicks a step towards you better run away right we make decisions instinctively not so cerebrally very quickly based on very small amounts of data and Kahneman and Tversky concluded that the true believer in the law of small numbers commits his multitude of sins against the logic of statistical inference in good faith without intentionally making that mistake it was exactly a small sample error that Tversky made 14 years later so there you go and here are the splash brothers when we did the study one criticism we got is that we were looking at the wrong players in fact Kevin Durant was the star of that series and so we looked at him and he didn't have a hot hand either and maybe with that I can can take questions have any studies been done on cold hands well it just seems intuitive that there would be such a thing as a cold hand you mean that it's more likely somebody's not feeling well somebody you know a player for whatever reason he's he's psychologically he's thinking about something else he's going through personal issues forgetting about the math behind it it seems intuitive that there would be such a thing as a cold hand so if a person if a player normally hits 50% of his shots and and we do see that because of psychological issues because of you know the playing with the flu whatever he's gonna have games where there is such thing as a cold hand then doesn't that mean that there has to be a hot hand to make up for that to get to the 50% so you're saying that the weighted average of the game averages has to be the season average I think right okay so this is that's a little bit different so one before I try to answer that I will say that we looked at a number of variants on the statistic that we used I don't happen to like that statistic I think we did it because we were asking a particular question in the original paper and also in our analysis we looked at other statistics as well like are there unusual runs in the sequence which is very related we didn't find any difference when we used different statistics but pretty much all the ones that we looked at involved conditional probability or run or what happened after that you're asking I think a different question you're asking if the shooting percentage is unusually low or unusually high is that the hot hand which is I think a question we we talked about and I don't remember if we looked at it very carefully but I'm actually looking at that question with a remarkable young man who is 15 years old and he's really awesome and we so to your point if Steph Curry's shooting percentage say is 50% no one expects him to shoot 50% in every single game but maybe we don't expect him to shoot half his game zero and half his games a hundred percent either asking a question about whether there's an unusual concentration of game averages in the tails instead of concentrated around the season average is a question we formulated and I have a few individual players but I think my friend Monov is running this for I think about 40 players over four seasons so we should have lots of lots of answers soon but I don't have one yet and I I do like that formulation of a hot hand I will say that we ran it for Clay Thompson right away because we always check clay he's supposed to be the streakiest player and he got all kinds of hassle early in the 2018- 2019 season for the first few games when he didn't play very well and people were throwing up their hands his career is over play can hit a three-pointer and then all of a sudden he seemed to snap out of it but in the data we looked at so far actually there was nothing unusual about Clay's distribution of averages compared to a random randomly generated benchmark we didn't see anything special in him we did find in Westbrook we found some extreme behavior but we're just getting started yeah of all kind right of all kinds I think this is related to what you just said but for this statistic if someone is a 10 of 10 out of 10 in the game they're not hot because the permutation stat statistic will give the same number with all the permutations as with the original statistic because one probability is zero and it's not a discerning statistic right right I mean that but but the question it depends on my question you want to answer I mean even though even though the statistics that measure these conditional probabilities are highly undercerning when you're watching that game you are sure something unusual is going on so this is this is the point of Tversky and Kahneman that our perception I mean I think I think you're right that this is just maybe a low power problem but the difference between what you can discern with statistics and what you feel with like your pulse rate are very different right and like it's related to that basketball is a situation there's a lot of shots but in something like football there's only a few passes or a few runs a game have you looked at these kinds of hot or cold hands over a season like our six game winning streets indicator of a hot team or not well I thought about it and I think even the original study that was done on longer strings but I don't really see how because how do you deal with the ends of the game I mean it's like I mean that this drives me crazy and financial economics data right the people with these continuous time models but what about nights and weekends you know that the market closes there's these big discontinuities and and it's somehow ignored in a lot of the studies so this to my mind I don't know if you think any differently kind of took account of those discrete chunks in which the data naturally came to us I would have made it more discreet if I'd had my way and had more data and not less discreet in order to try and discuss the phenomenon I'm wondering so you used randomly permitted strings to to bootstrap distributions for your T2 statistic I'm wondering how the results change if you use just completely randomly generated strings such that the probability of success is the same we did that didn't matter thank you thank you this is a great talk it gave me a lot to think about I don't follow basketball at all so my entire knowledge of the subject is now from your talk but a female your talk was the tension between the mathematical academics and the practitioners empirical observations and I'm wondering from listening to you if you're even asking the same question because you were looking you kind of framed your talk as curry and Thompson and then you were also looking at the warriors in general looking at a specific player and seeing their streakiness but in the beginning of your presentation I think you quoted a news article saying that oh now Bill Bradley has the hot hand right so that seems to me that they're thinking of the hot hand is something that exists and pass from player to player so if that's the case then yes somebody's always going to be statistically ahead and you were saying that that person gets passed to more often so they're probably their overall scoring is going to be higher even if their percentage isn't but then if they don't well though they pulled off and now the hot hand is over here so I'm wondering if if the academics and the practitioners are even asking the same question when they're talking about the hot hand well that's a good point and I rarely find academics and practitioners talking about well I rarely find academics and academics talking about exactly the same thing but I do if you we did look at specific games where there was clearly kind of a high high intensity going on and this statistic like in the one with clay with 60 points I think that that the commentators many times during that game probably accused clay of having a hot hand and this statistic did not do that so so I your point is taken but I don't think it negates the subject we presented here. Hi you actually touched on my question when I heard the first time hot hand I was thinking of when I grew up in I grew up in New York City and we played 21 and you were talking about a more discreet streak so when you I saw that string I was wondering why you would just call you could put a string like that because to me in basketball you shoot and if you make it again right then right after that shot as opposed to just a whole 60 do I still play 60 minutes I mean they keep changing the rule so that the discreteness to me would be actually almost can't be compared because with a field shot you get one or two shots so then I'd be more impressive somebody gets it in and gets it in again as opposed to within a quarter they make a lot of shots but you don't know what happens in between am I making my question clear completely to me it's a slightly different statistic I'm interested yes how many who can hit who can make 60 points is really good but did he make it all in one quarter did how much space did he have time to me is a more has more of a yeah yeah has more has more to deal with the hot handedness then just looking at the totality so that is a really wonderful question and when I gave this talk at the Berkeley Institute for Data Science I got the same question from a data scientist who wanted to take account of the temporal effects how quickly the shots came together and so on and that I think would be very very interesting and given the incredible wealth of basketball data on the web I think that's something we could do kind of anytime so great question but I don't know and it would have to be formulated like we would have to make a new statistic we would have to think about it let's give your speaker one last hand