 I'm going to turn it over to someone who is a serial entrepreneur. He'll tell you just what that means. He's going to kick off tonight's program. Please join me in welcoming David S. Rose. Good evening and welcome to this full house for a really exciting evening. My name is David S. Rose. I am a serial entrepreneur and serial angel investor. What does that mean? That means I'm somebody who starts companies. That's what I do for a living. I really love doing it. I've been doing it. I'm actually a fifth generation serial entrepreneur. I was a finalist for the entrepreneur of the year award back in the dot-com boom. My father won it in 2003. He's now 90 years old. So it runs in the family. Being an entrepreneur these days means taking advantage of all kinds of new things and new ideas and particularly new technologies. And new technologies increasingly are based entirely on math, which makes you wonder why you're sitting here. There are several ways you can look at anything, any type of human endeavor by different kinds of analysis spectrums and characteristics. Let me give you four of them. One of which is, is it useful? Is math useful? Well, historically back from the ancient Greeks who had more mathematicians than they had plumbers back then. In fact, they think they had more mathematicians than they had anything else, including farmers or sculptors, since much of what we know about mathematics stemmed from the ancient Greeks. Mathematics has been not only an intellectual form of endeavor, it's been a utilitarian form of endeavor. It's how we build things. And as you go through history, you get to the Enlightenment, the Industrial Revolution, all of a sudden, manufacturing things, building things, designing things, engineering and eventually physics, these are all based on math. You then get into the 20th century, technology accelerates, and all technology ultimately is based on math. So is math useful? It really is. Has anybody here heard of something called Singularity University? Raise your hands if you have. Whoa, about five people in the audience. Okay, cool. Singularity University is a sort of think-tank postgraduate program in San Francisco and California based on the work by Ray Kurzweil, a book called The Singularity is Near, about how technology has been advancing exponentially, exponentially since the beginning of recorded history. And that means you all, maybe you have heard of Moore's Law, which is the number of transistors on a chip was doubling every 18 months. Well, what Kurzweil showed in his book is that technology as a whole, the pace of technological advancement was doubling every 18 months since the beginning of recorded human history. So you would look at me and say, well, that doesn't sound right. I mean, then how come Neanderthals didn't have iPhones? And the answer is math, because when you go exponentially, you're doubling and doubling and doubling, and it's like the old game of grains of rice on a chessboard, things get really big, really, really fast. And so today, increasingly, technology is playing a vital role in everything we do. You've seen the price of things dropping to virtually free, you've seen people being put out of jobs because of everything from automatic, you know, ATM teller systems. Back when I was in high school, I went to the Smithsonian in Washington, and there we saw an exhibit of this robotic teller, this automatic teller machine that someday would be in banks where you could actually put a card in and get money from a machine. That's how old I am. And so now, of course, tellers, what's a teller? Anybody seen a teller recently? Have you seen a travel agent recently? Increasingly, you're not going to see retail clerks or truck drivers. And all of these major impacts on our society are coming from math, from technology and automation. And therefore, if you're not part of the solution, you're part of the problem. And that's why STEM, science, technology, engineering, and math is so critical for everybody to understand these days because this is the fabric of our society. Okay, math important? Yeah. I guess math is important. Well, the next question is, are you any good at it? Well, our speaker tonight is somebody who is particularly good at it. This is somebody who's got a degree, two degrees, three degrees in math, all the way up to a PhD. And a lot of people from whom we'll be hearing is somebody who knows math and has put it into practice. And there are theoretical mathematicians, academic mathematicians. He is an applied mathematician, that's what I would call him, who has taken his study and knowledge of math into the realm of business and work. He is one of the preeminent mathematicians involved in creating all the things you've seen. Alon has worked at Google. He's worked at Facebook. He started his own company. He worked for the Israeli Armed Forces. He's now working with Intuit, which acquired his company. And in all of these things, he is one of the key people who knows how what you do online, ultimately, is affected by math to see what other people will do to you, for you, with you. What ads are shown to you, what things you see in your feeds, all those kinds of things. So this guy is at the top of his profession. The third thing is, do you like what you're doing? There are some people who are very good at doing something, but really don't like it. They see it as a job to put food on the table and that's it. Well, in this particular case, our speaker tonight loves math. Who in this audience loves math? Ha! 100%. That's why you're here because duh, if you didn't like math, you wouldn't be sitting for a countersend math. So yeah, well, welcome tonight to somebody who is of your ilk, somebody who truly loves math. And this is fun in addition to being work, right? And so, as Aristotle said, earlier in our previous introduction, I quoted the ancient Greeks when I was appropriately upgraded, saying it was specifically Aristotle, that happiness is the exercise of one's vital powers along lines of excellence in a life affording its scope. Exercising one's vital powers, these are powers that Alanimid has in terms of math and he is exercising them along lines of excellence in his chosen profession. And that makes him happy and he loves math. And then the last question is, can you teach it? There are people who are very good at doing X, Y, or Z, but they can't teach it. They just love doing it, sitting off in a corner doing it for themselves. Or they try and teach it and don't do a good ID, don't do a good job of it. In the case of Thai speaker, Alanimid is one of the best teachers of math around. And that means he inspires people and helps them understand what it's all about. How many people here have heard of a website called Quora, Q-U-O-R-A? Oh, right, right, that's good. So a lot of people here actually haven't heard of Quora. Quora for many of us is an addiction. For those of you who raised your hands, how many are addicted to Quora? Any of you here? Oh, a small bit. Okay, so that's very interesting. You are the vast majority of users who find Quora an interesting subject. Well, some of us happen to be truly addicted to Quora. And that's actually where Alan and I met, because we are both total Quora addicts. And to put things in perspective, Quora has over 300 million monthly unique users who come onto the site asking questions, looking for answers. And of those, Alan and I are two of the top 50 most followed people on the entire site. We've actually got 90,000 followers each. His are people interested in math, mine are people interested in weird subjects that I write about. But they turn to Quora and to his answers because he is a wonderful humanizer and a popularizer and explainer of math. And in his real life, in addition to his job, he's working with a school for kids who love math. It's a math-based school for kids. And so I can think of no better person with whom you should have an encounter with math than Alan and me. Thank you, David. And thank you, Cindy, for inviting me over. Thank you all for being here. Thank you, Paul. Probability paradoxes is what we'll be talking about today. And I know that the words may seem unfamiliar or complicated or intimidating. Please, don't worry. It's going to be a very... Oops! Everything is going to be okay. I have a little confession. The reason I snuck in this little shenanigan, there's actually a reason for it. People who do this come to talk about math to kids, to adults, to teachers, to general audiences. A lot of the people who were here in math encounters before when you were here over the past months and years, one of the things they have in common is they come and they try to make math seem less scary, seem more approachable, seem more doable. Because if anything, many people are more afraid of math than we would have liked. So we take a complicated subject. There was a talk here recently about four-dimensional space and general relativity, and we try to make it approachable. We try to make it seem friendlier. Tonight is different. My job tonight is to do the opposite. I'm here to scare you. So be afraid. Be very afraid. The reason I do this, and I know I'm just a little bit joking, but the reason I do this is of all the fields of math, probability theory has a strange and unique distinction of seeming easier than it actually is. Very few people misinterpret multivariate calculus and think that it's easier than it actually is. Usually if anything, they are afraid of it and think it's harder. Probability theory is not like that. It's a familiar subject. We all think about and talk about probabilities in many different ways. And when the question comes up, I hear people often say, oh, probability theory was like the easiest course in college. It wasn't abstract. It wasn't scary complicated. It wasn't very technical. It was pretty straightforward. I think that perception is incorrect, and I'm here to correct it. So if you leave this room today knowing less probability theory than you did walking into here, I'll be very happy. I will have succeeded. That's my goal for tonight, is to un-teach you what you think you know. Shall we? Let's start with the words, probability paradoxes. What do they mean? We have two words here, so at least we should start by making sure we all agree on what they mean. The easier word here of the two is... Paradox. Paradox, of course. What's a paradox? What is it? Just shout it out. It goes against the established knowledge. It goes against the established knowledge. Something impossible. Right. Contradictions are something impossible. Something that contradicts its own confidence. Something that contradicts itself. Those are paradoxes. In mathematics, we try very hard to stay away from genuine paradoxes. As far as we know, there isn't one in modern math, and I won't be showing you one today. The paradoxes we'll be talking about today are closer to counter-intuitive or confusing or doesn't seem right. Those are not genuine logical contradictions. Those are just things that feel wrong and defy our intuition. That's the easiest word. What about probability? What's that? Again, come on, just say it. Chance. Chance, yes. What else? Measurement of likelihood. Measurement of likelihood. When people say chance, I ask this question often. I say, what is probability? Almost always the first answer is chance. My next question is, you can guess it. What is chance? What is chance? The answer to that is almost always likelihood. I think it's going to be easy for you now to guess my next question. What is likelihood? When something is very likely. I'll have a little confession. I actually have a slide here with your answers, because it's usually the same set of answers. People have a lot of words for probability. There are odds and ratios. Probability can be interpreted in an objective fashion, in a subjective fashion. It has an interpretation in terms of degrees of faith, or how much we believe something to be true. There are many, many different interpretations. Let me start this evening just by a baseline, making sure we all agree on one thing. What is probability? Real answer? Nobody knows. That's the most honest answer I can give you. And that's the most honest answer I think you'll find anywhere. Nobody knows what probability is. There's many different interpretations. There's many different schools of thought. The schools of thought disagree with each other, because that's what schools of thought do. And if you distill everything that's been written, and a lot has been written over the past 150 years, about probability, if you try to summarize it all in one sentence, is we don't know. One thing that people do seem to agree on is that this ratio thing that you mentioned, probability is a ratio, ratio of what? Ratio of two counts, the number of favorable events, divided by the number of total events. I assume that many of you have seen something along these lines. I'm studying about probability in school. That's a very famous interpretation of probability. Sometimes it goes by the name the frequentist interpretation. And a lot of people, myself included, are not really buying into this definition of probability. And I wanted to demonstrate why I don't buy this. So yesterday, I just went on Google and looked for probability in the news. And immediately this popped up. Let's read it together. Brexit delay probability raised to 85% from 60%. J.P. Morgan says. If you read this sentence on the news, do you feel it's strange or weird or you don't know? I don't think so. We all read the sentence. We basically understand what J.P. Morgan is trying to say. But can we interpret that sentence in those terms? A count of favorable things divided by the total count of things? Well, first of all, Brexit, favorable, I don't know about that. But let's even assume that you're okay with that. What exactly are we dividing by what here? When we say the likelihood of a Brexit delay used to be 60%. But now J.P. Morgan has calculated it to be 85%. Those ratios, 60%, 80%, 85% are ratios of what exactly? Is it if we had 1,000 Brexits, 60% of them were delayed, but now 85% of them are delayed? Where are those 1,000 Brexits happening? And how did J.P. Morgan measure them? How many Brexits have already happened? No, no, no. They're saying it's supposed to end October 31st. So what J.P. Morgan is saying? It may end November 1st. It may end. We all know it may end at all kinds of dates, but they're saying something a lot more specific. They're saying that the chances that it will get delayed is now 85%. 85% is the ratio. It's a number. 85 divided by 100. And I'm asking, hey, J.P. Morgan, can you tell me, what did you divide by what to get this figure? If we throw a roll of die or flip a coin, we know what we're counting, but what exactly is being counted here? What's being counted here, if you ask frequentists, are some kind of imagined universes where Brexit may or may not get delayed and we somehow surveyed those universes and made an assessment on how many of them are going to get delayed. Nobody actually did that. So that is why I'm not super keen on the frequentist interpretation. That's why many other people are rejecting this interpretation. This is just one example of why those things are difficult. They're actually also meaningful, as I'm sure you know, probability theory is not just a game of numbers in math. It actually affects the very fabric of the universe. For about 100 years now, quantum mechanics, the basic theory we have to describe how the universe operates, is probabilistic in nature. So physicists now, among other people, have to wrestle with the question, what do we mean when we say that the particle may or may not take this path and the probabilities of them doing so are such and such. This quote I have here is from an actual paper by an actual physicist, David Wallace, who wrote a lot about quantum mechanics, and he's trying to portray that quantum physicists talk a lot about probability without ever explaining what it is and when you ask them what is it, they don't know. So he is taking one of the basic rules of quantum mechanics. If a system is in a certain state, there's a certain way to calculate its probability of being in a certain eigenstate. And if we replace this word probability with squirtle flunkicity, it's the same sentence, but do we understand it now better or worse? It seems less clear, but why? That's because we know what probability is. Really, says David Wallace. What is it? We don't know. There is a frequentist way to explain quantum mechanics, you may have heard of this too, the many-world interpretation, where we're assuming that our universe splits every moment in time to all the possible things all particles could do, and all those particles are doing all of those things in those universes in different proportions. That's a nice way of thinking about things. Philosophically, it's a little bit scary. The number of multiple universes is really, really scary. But I'm just showing this as an example just to show that this is very recently people as smart as quantum physicists are still debating what do we mean when we say probability. We teach it to all students, but we never explain what it is. So this is all complicated and confusing, but there's also good news. We are mathematicians. We don't care. We don't care what anything is. We have axioms. The axioms tell us what the thing does, and that's all we need. I want to remind you, those of you who studied geometry, geometry talks about points and lines and circles and triangles and angles and all these different things. One of the very first textbooks that was ever written about anything was written about geometry. This was many years ago. The textbook was... Sorry? Euclid's Elements. And in Euclid's Elements, if you read it, one of the first things Euclid does as anyone should do is define his terms. I need to talk about points. What is a point, says Euclid? He defines a point. Do you know how he defines a point? As no breadth or width, that's a point. And how does he define a line? A line is that which lies evenly with itself. I think it's beautiful, but I don't think it defines anything. Do you feel that those are good definitions for a point or a line? That which lies evenly with itself. Euclid didn't think so either. So what did he do with those definitions in the rest of the elements? Nothing. He never once goes back to those definitions. Euclid never says once in the Elements, look at this diagram, the thing we see over there is a line because it lies evenly with itself. He never says that, because he knows full well that this doesn't mean anything. The beautiful thing Euclid did, the thing that affected human thought forever, was set this aside. I'm going to declare a few axioms that I'm just asserting that points and lines satisfy, and I'm going to derive everything from those axioms. That's all I need. And for 2,000 years since then, that's all mathematicians did. We define our terms not by telling anyone what they are, but what they do. Here are the axioms they satisfy. In fact, one of the reasons geometry and probability theory is that the points and the lines we talk about doesn't need to be points and lines at all. There are many fields in math where we find ourselves talking about geometric notions without any geometry being present. The points can be giraffes, the lines can be groups, all kinds of things. In probability theory, we have axioms. I won't go through them. They're very familiar in an everyday kind of way. Probabilities are real numbers. They're between 0 and 1. When you add up the probability of certain things, you get the probability of the union of one of those things happening if they're disjoint. All of those things are pretty straightforward, and that's all we need for tonight. So tonight we won't actually explore the question of what is probability. We'll talk a lot about probabilities, but we won't care about the philosophical underpinnings, only about the math. So, everything is fine. After all. The first story I want to tell you is called Tuesday Girl. This story begins with a puzzle which has become very, very famous. I suspect many of you have seen this puzzle before. What I'm asking all of you to do is bear with me if you've seen the beginning of this puzzle because you may not have seen its end. There is a twist ending there that I'll try to walk you through and we'll see how this goes. So, the story is very simple. You meet a person, a person you don't know, and in the course of the conversation the person mentions that they have two kids. And we're wondering what is the likelihood, what is the probability, what are the chances that those two kids are both girls. Now I'm going to take a quick pause here because I'm going to be talking about gender. Gender at birth, and babies, and boys and girls. And I want to acknowledge the fact that we all know that gender is more complicated than simple boy-girl distinction. We agree with that. I'm not making any political statements here. I'm simplifying for the sake of a simple math model. So, please, I just want to make sure nobody's offended. It's important, it's important. We're talking about boys and girls as a simple model for something that happens with likelihood 50-50 50-50, independently. So, please assume that, if that's okay. So, if a person has two kids and that's all you know, what's the probability that they have two girls? What is it? One in four. You guys are amazing. One in four. And that's very simple because the combinations of having two kids or having two girls or two boys or a girl and then a boy or a girl and again, I'm simplifying. No twins, nothing weird. Just very, very straightforward. Two kids, two girls. Okay? Okay, so far. Now, the next part of the story that you still may be familiar with is you meet another person kind of look the same but work with me here. You meet another person, random person and they also say that they have two kids but then at the course of the conversation you mention something about their daughter, Jane. Completely at random. Now, you may be thinking maybe they had some hidden agenda. Maybe we are in a city where everybody only talks about their daughter if they have one or something like this. So, again, let's simplify. To make this completely concrete, imagine that this is simply, we're looking at the US census of families. We're looking at those families that have two kids but we are restricting to those families that have at least one daughter. A person who has two boys cannot mention their daughter, Jane. Everybody else can. So, we're assuming that we're simply sampling from this portion of the population. Now that we know this, what are the chances that our new acquaintance has two girls? Now that you know that they have at least one, what are the chances now? All right. 50-50, raise your hand. One third, raise your hand. Awesome. Something else, raise your hand. Okay. We have some original thinkers here. I'd rather not talk about other child because we're not even sure they have a daughter and the question is what are the chances that they have two daughters? That's it. We don't know if Jane is the older or the younger. We don't know anything about her. All we know is that this is a random family from the cohort of families that have two kids and at least one daughter. Okay? And the answer really is one third, not one half. And the reason is this. These are the same cohorts we were looking at a second ago. Two girls, two boys, girl boy and boy girl. We eliminated a portion of this population as irrelevant. The people who have two boys are not in the game. Right? They can't mention a daughter Jane. From the people who are left, everything is uniform. We don't know anything else. All we know is that there is a Jane. That's all we know. Right? If it's easier for you to think about a million families here, a quarter million in each one of those cohorts, we've eliminated this cohort. We're left with three-quarter of a million families of which 250,000 have two daughters. That's a third. If you pick a random family from this cohort, one in three will have two girls. So far, so good. This is really fairly old ground by now. This is fairly familiar. If you had gone to interview at some analytical company over the past few decades, there's some chance someone would have asked you this question. It's become famous. A basic test of conditional probabilities and can you calculate basic things and not get confused? But a few years ago, someone called Gary Foshee in a Gathering for Gardener meeting upped the ante on this in a beautiful way. So here's the story. You meet a person. Once again, they look like everybody else you've ever met. I don't know why, but you meet a new person. And they also tell you that they have two kids. They say something a little bit strange. They say, I have a daughter born on a Tuesday. Why would someone say something like this? I don't know. But it happened. So once again, to make this completely clear and unambiguous, we're looking at the census of U.S. families and we're restricting our cohort just to those families that have two kids of any gender, at least one of which is a girl born on a Tuesday. Okay? If I handed you the U.S. census you could have done this. It's just a matter of running a query or filtering in Excel or whatever horrible tool we're using. Right? Now the question is of these families how many have two girls? What are the chances that this person has two girls? Is the answer one-half? Is the answer one-third? Is it something else? I'm going to tell you what the usual reaction is. I know that you guys are smart. So I'm going to tell you what most people say at this point. Most people say Tuesday what does Tuesday have to do with anything? We just answered the question about having one daughter and the answer was one-third. Now we know that the daughter was born on a Tuesday. Big deal. She had to have been born on some day of the week. Right? So the common answer to this question is that the answer is still one-third and that's wrong. It's very wrong. In fact the answer is almost one-half. The answer is this. Just the number you would have guessed right away. Isn't it? Right? Now actually figuring this out if you just sat down the counting is not very difficult. Let's do it together. It's not hard. These are all the possibilities. These are all the combinations of families. Think of this as the first child. The first child is a girl or a boy. The second child is a girl or a boy. Each one of them was born on some day of the week. This family for example has a first daughter born on a Wednesday and a second daughter born on a Tuesday. I'm simply surveying all the possible combinations of genders and days of the week. Now many of those families were eliminated from our cohort. We're only looking at these families. What are those families? Those are the families that have a daughter born on a Tuesday. Either first child or second child. Do you agree? It's just a matter of checking which cell has a daughter born on a Tuesday. That's our cohort. This is what we're looking at. Of these families these are the ones that have two daughters. Now just count. 1, 2, 3, 4, 5, 13 are in there out of 27. That's it. That's your answer. Favorable number divided by total number. That's it. Now that seems strange. The person told us they have a daughter we are assuming, we are assessing their chances of two daughters as one third. The person told us they have a daughter born on a Tuesday and are currently changing our assessment of the chances that they have two daughters. Why? One way to think about this is to go back to the first question. If someone says I have two kids and my oldest is a girl what are the chances that they have two girls? Now it's easy. That's 50-50. Because now it only depends on the gender of the younger child. By telling you about the day of the week they kind of almost pinpointed it's almost like saying my oldest daughter is a girl. The upshot of all this, I want to move on but just to kind of the sound bite to keep in mind is that when you're looking at samples and slicing them in different ways you could get very strange biases. If anyone here knows a data scientist or an analyst or any sort of data worker you'll hear them saying slicing and dicing at least six times a day. That's what we all do. We slice and dice all day long slicing and dicing the data. This was an example of slicing the data. We took the US population and we sliced it just by those families who have at least one daughter born on Tuesday. It may sound like something strange to do but believe me stranger things are being done. When people are exploring the effects of A on B they might be looking at a family where at least one of the parents is Native American for example. So that's not an unusual thing to do. But when it's done your assumptions on what is going to happen to your probabilities may be wrong. Next story. When the part disagrees when the parts disagree with the whole when the whole disagrees with its own parts. I want to share something that happened just a few years ago. It's a very simple tale but I'll try to convince you that it's actually significant and happens quite often. So there was an article published by a couple of times a few years ago with a study of how wages are behaving over time in the US. Are we making more or less money than we used to? They did a very careful analysis inflation adjusted and everything else you might have wanted from the study and they looked at US median wages between 2000 and 2013 and they found out something very exciting and positive. US median wages in those 13 years went up by almost 1%. Adjusted for inflation. That's good. Should be happy. Somewhere later in the same article they did a slightly more detailed analysis looking at the same figure broken down by level of education. So they looked at people with no degree people with a high school degree people with some college degree or some college education and people with an actual undergrad degree. And so it's a good thing to look at. If we know that wages are up in the US overall we're curious to see how this number changes when we look at specific education levels. And they did it very very well. Everybody in the overall study was in one of those cohorts nothing was double counted, nothing was missing very very well. And here are the numbers. For example, for people with no degree over the same period of time the median wage actually went down. If you can't read the number it's okay just notice that it's red. For people with a high school degree wages also went down. For people with some college degree wages also went down. Now I'm going to stop here for a second and can I have you guess can someone just guess what do you think is in this last rubric there? Green. Right, green. And it has to be pretty significant green. Why? Because we need an increase overall in US wages and we need to counteract that terrible downward trend from people without a college degree. So here is the number. Now the funny thing is, the funny thing is the very editors of the New York Times the people who wrote the article they didn't even notice this other table tells a very different story and they got some angry letters from readers who are like are you kidding me? You said that US median wages went up over those 13 years and now you're telling me that they went down for people with no high school degree, down for people with a high school degree, down for people with some college degree and down for people with an undergrad degree. So who is it exactly the person, the magical person who saw their wages go up? And I want to be clear everybody is in one of those rubrics. We didn't leave anything for Silicon Valley. Silicon Valley people are all here. No, for real. And if you think there's something wrong with this data, you're wrong. You don't know how many people are in each group and you don't know what the cutoff point is for you. The cutoff points don't actually matter but you're right, we don't know how many people are in each group and that's the factor that makes all the difference. Because the thing that actually happened in the US over those years was that people became more educated and people with higher education do earn on average more. So the thing that changed wasn't the rise in the metric it's not that employers have become nicer they actually haven't they became more mean but they found themselves having to pay more people with college degrees. So that may seem reasonable it's not a paradox but it's still called a paradox. It has a name, it's called Simpsons Paradox and I would encourage all of you to go look it up on Wikipedia not now, but here's the thing about Simpsons Paradox I think it's important and I think it's happening a lot it's happening there's examples, if you look it up you'll find examples of Simpsons Paradox in economics you'll find examples in sports we measure the quality of athletes by all kinds of metrics Americans are really good at that field goal percentage RBIs and all those metrics and you can have very strange things happen you can look at at a period of years you can look at two athletes and you can observe that in each and every year, athlete A had a higher RBI, higher metric from their colleague but if you look at the entire time frame it reverses so A was better than B in 2015 and 2017 and 2018 but over the four year period, player B had a higher RBI I'm not joking, it can happen it actually happened it happens in business, it happens all the time I'll get to the story in a moment, but here's what I want to share this happens more than people realize for two reasons one, they don't realize two it happens even if you don't notice it and here's what I mean by that if you look back to the like an honest question if I showed you this New York Times data and I showed you the median wage in the U.S. went up and then I told you it went down with here, here and here but it went up for people with bachelor's degrees you would immediately conclude that it's those people who are driving the higher wages and that's wrong it's just as wrong as it is with the actual data the thing is, Simpson's paradox rarely slaps us in the face with something so completely contradictory but when it doesn't slap us in the face doesn't mean it is not there a metric can move because of the shift in populations and not because the metric moved or it can move in the completely opposite direction story time no slides, okay just listen 2008 I was can I say I was a young product manager I wasn't very young but I was a fairly fresh product manager working at a search engine company whose name is a misspelling of a really big number I won't share the exact details it was 11 years ago I don't think there's anything secret there but I'll also simplify things a little bit but this thing actually happened there are people in the audience who can attest to that your Google, imagine your Google you want to know if your business is doing well what are you looking at you're looking at revenue, revenue is a good metric yes we all love revenue but revenue is not enough revenue can go up and revenue was going up for Google in those years because people use Google more and more how does Google make money? ads you run searches on Google, you get search results and you get ads now here's the mysterious thing Google only makes money when you click those ads Google makes no money the mysterious thing, maybe that's today's greatest paradox is raise your hand if you click Google ads nice thank you maybe this is New York so like four people are timidly raising their I live in Silicon Valley when you ask this question in Silicon Valley nobody does and that's a very mysterious thing because every time a Google ad is clicked Google makes a few pennies now on the one hand nobody clicks the ads on the other hand Google makes $50 billion a year so it's really amazing in reality people click those ads a lot maybe they don't know it or maybe they don't confess it but they do and Google makes money from those ad clicks and one of the metrics Google really cares about is revenue per search how much money are we making from each search being done why is that a good metric because it says look there is a search team it's making searches better and people search more and more but they're not about making money there's an ad steam this guy who's all about extracting money from those searches and if revenue goes up it could simply be because people are searching more and more and more that doesn't mean that the ad steam is doing his job well at all so Eric Schmidt CEO of Google at the time everybody cares about RPMs the CFO cares about RPMs Larry and Sergey cares about RPMs 2008 was not a great year for the US economy it was also not a great year for AdWords so again I'm simplifying here it wasn't a great year it was doing fine it was still growing but let's pretend for a second that RPMs were really going down like you're running a business and your amount of money you're making per search is going down you're worried one of the things you do is slice the data you call for example your different country managers Google is doing business worldwide right there's North America, there's South America, Latin America there's APAC, there's AMIA there's all kinds of regions in the world and you call the country managers and you ask them tell me how are you doing how are your RPMs looking since last year and everybody says everything is great everything is fine North America says RPMs are up Europe says RPMs are up so says Asia and yet miracle of miracles, RPMs are down if you're a confession, like seriously, you're the CEO what do you think someone is wrong or someone is lying neither of those conclusions is correct for the same reason we just looked at and this actually happened in those years there were certain countries who were growing very fast 2008 searches Google searches in the US were still growing but not very fast everybody in the US was already using Google not everybody in India was using Google and the difference between 2007-2008 in terms of usage of Google was skyrocketing however, RPMs in India are very low clicks in India are paid for by rupees very small amounts the RPMs in India were growing the business was great but the significance of India as a fraction of Google's business was becoming more and more prominent which dragged the RPMs down now the thing I want to convey here is that as a business decision maker this put you in a very weird position you have a metric which seems great revenue per searches but it's going down for reasons that have nothing to do with how you actually monetize your searches you're monetizing your searches very well in every country when I have this conversation with data scientists and analysts we have a long conversation about what can we do about what should we do there's all kinds of suggestions to look at the different countries and the problem is that country is just one way of breaking down your business there's usually any metric like this is comprised of dozens of different ways you can slice it by so what do you do with index some of you may have heard about the idea of an index where we freeze the proportions the nice stock market index is like that and people criticize nice for this but that's actually a good idea for that reason so this was a quick tip about Simpsons Paradox I hope I've convinced you it's happening a lot and I hope I convince you it can be very confusing even for someone who knows it it can be confusing I've known about it for many years and I still get confused I want to share another instance when we were discussing probability situation it has to do with a hot hand the hot hand fallacy now I gave the same talk a few hours ago and people pointed out to me that there was a talk here a few months ago about the hot hand fallacy so I'll try to be original I'll try not to repeat everything that was said I think maybe my perspective on this is a little bit different you tell me the question the basic question is a very very natural question to have is there such a thing as a hot hand a hot hand is a person let's say an athlete has a good streak they just sunk four, three pointers in a row they're very excited the audience is going wild they're happy they're going to do better their next throw is they're going to have a higher chances of making it than normal that's a hot hand for many years people talk about hot hand as if it's obvious you will see those amazing nights where an athlete comes to a game and just knocks it out of the park it's got to be something called a hot hand and then in 1985 a paper was published that created the term the hot hand fallacy this paper was a very very successful paper shall we say it got cited and quoted all over the place and it demonstrated very methodically and very thoroughly that there is no such thing as a hot hand how did they do this they looked at actual NBA games and counted the number of times someone made a streak and then made another one or made a streak and then failed they even went to a Cornell and took actual athletes men and women a few dozens and had them stand and throw throw throw throw throw and counted their shots and try to see if they're showing any signs of a streak of course they had to analyze data they analyzed data very thoroughly I'm talking about pretty prominent people Amos Tversky who is listed here as one of the authors he was the research partner of Daniel Kahneman who went on to win the Nobel Prize in 2002 Tversky himself did not win the Nobel Prize because he wasn't alive anymore he died six years earlier but everybody assumes that had he been alive he would have been he would have shared that Nobel Prize with Daniel Kahneman we're talking about someone very knowledgeable very successful they ran this paper and a new thing was created the hot hand fallacy now it took many years for decades this thing stood and as you may have heard if you were here in May to hear the previous talk about it it got debunked just recently in many different ways some of the ways were about the statistical significance of those results people claimed that they just didn't look at enough data hot hand if it exists can be very subtle it makes you just a little bit better in your fourth throw after a streak so maybe they just didn't look hard enough or didn't gather enough data there was a paper in 2011 that claimed that they can see evidence of a hot hand by looking at 300,000 games or throws I forget a lot of data but I want to talk about a different paper which said actually there is a hot hand and the way they did this over here in this paper is actually interesting because what they found was that there was a genuine mathematical mistake a very natural one but still a mistake in the original paper of Tversky et al I'll quickly explain what it is again sorry if you've seen this before even if you have maybe you've forgotten so please tell me you've forgotten and I'll do it again let's say we're flipping a coin and if I ask you how many heads you expect to see in those ten flips you'll say five I'm not out to get you it's okay it's five when you flip a coin ten times you expect five heads that's okay I'm not claiming that this is wrong but we're not looking just at heads we want to see if there's any evidence of a hand or hot hand so let's say we're doing something very simple we're looking at the times we got heads think of heads as making a throw dunking it right a head is successful so I want to see what happens right after a heads did I get another heads or did I get a tail so I'm just counting after the first head I got a tail after the second head I got a head after this head I got a tail here I got the heads here I got a heads over five instances in this particular sequence I got three heads out of five sixty percent here of making a heads right after having a heads sixty percent here and I do this let's pretend I do the sequence of ten many many times every time I get a sequence of ten I can count the ratio of finding a head right after a heads or not these numbers sixty percent fifty percent are simply counting those heads that came after heads when you throw a head is there a difference between your chances of getting a heads or tails right after in the flip of a coin coins have no memory like water water has no memory coins have no memory the fact just through a heads doesn't impact the next throw right so clearly we expect fifty percent heads versus tails right after a heads that's wrong that's incorrect if you do this carefully you do the math which nobody did for 33 years until 2018 I'm talking about the paper that was published last year you'll find that the probability of a heads right after a heads in this limited set of ten throws if you repeat it is actually slightly less than fifty percent if it's slightly less that means that when you're observing in real life people throwing in basketball and they do make fifty percent of the shots right after a actually doing something which goes way beyond the norm was the null hypothesis in Tversky's paper they said they wrote it in the paper they said we they didn't even say we assume their metric was making the shot versus not making the shot after a streak of three and they said obviously under the null hypothesis this should be fifty-fifty and it's not it just isn't and again this is not a methodological mistake it's not p-hacking it's not anything like this it's simply an error in the math the easiest way to observe today how the cultural attitude has changed around the hot hand fallacies to go to Wikipedia and look for hot hand fallacy you know what happens you get redirected to an article called hot hand it's no longer called the hot hand fallacy because it's no longer a fallacy the consensus nowadays is that there is actually such a thing I want to spend some time with the closing of this to talk about something else not about the math why should we care why does all this matter I think it does and I want to share a few thoughts about why it does I want to start by doing something a little bit mean I'm going to single out a person and I'm kind of going to crucify that person but I I'm not doing anything that's not way out there in public this is all nothing here is personal I don't know Brian but he's a real person and he was a pretty important person in some ways to all of us he was the executive director of the USDA center for nutrition policy and promotion which is a long way of saying he got to tell you what to eat I'm not joking this is a center that has a lot of decision making power about our food ingredients and recommendations and what's okay and what's not okay he was a director of the food and brand lab at Cornell and he was holding an endowed chair in the applied economics and management department at Cornell he was a prominent researcher you notice that I'm using the past tense he no longer is he last year again this is just something from last year Brian one thing was investigated for having a range of errors and misconducts but also genuine errors in his published papers and he had to issue an apology of sorts saying that he's sorry for his mistakes but other than an apology he was also dismissed from all of the positions I just described he's no longer with the USDA with Cornell he's actually no longer a researcher now the reason I picked on him so to speak we've all heard stories about scientists who have done awful things copied results from one paper to another I'm sure you've seen these stories the first time anyone noticed that he was doing something wrong was because he blogged about it proudly he bragged to the world about his achievement and what he did to get the results he wanted from an experiment and when people read this they were like wait a minute that's not right and what I'm trying to convey here is that Brian didn't even realize that it wasn't right and you know that it's not right because there's an XKCD about it you can look it up it's a known XKCD people say there's no relationship between green beans and acne maybe there's a relation between different colors of beans jelly beans and acne so you're on an experiment and you're looking for a correlation between jelly beans and acne you can't find it but then you're saying okay let's look at the red beans no correlation, black beans, no correlation yellow beans, no correlation, purple beans green beans, whoops slight correlation the next thing that happens is there is a sensational headline green beans jelly beans cause acne why did that happen? when you have enough data, if you slice it in enough ways something's bound to happen it has a name, it's called p-hacking that's what he did and he didn't even realize that was wrong again I don't mean to pick on him in particular because look what happened in 2016 there is an association in this country called the American Statistical Association it's the largest association of statistical scientists in the country and in 2016 they did something which I gotta say I've never seen any professional association do anywhere in the world ever they admonished everybody to know their stuff the meaning of p-values should be clear to any student statistics the trouble is that people who use these things are not trained statisticians or professional statisticians they are scientists from all walks of science social scientists, psychologists economists, nutritionists everybody is using p-values and most people don't understand what they are that's a very strange situation it's like taking every PhD in this country and telling them guys you forgot what you learned if you learned in freshman year it's not done but it was done a few years ago because people abuse the stuff all over the place and they don't realize that they are doing anything wrong the stuff I talked about, Simpsons Paradox and the subtle biases you can get from sampling and let alone this very unusual effect that you get from short samples of bursts of what happens with independent events all these things are way more complicated than this p-values are basic and here you have the larger professional society issuing a statement teaching organization but they had to teach that's not right people something is really wrong here probability theory the foundation the basic mathematical underpinning of a science called statistics well it used to be called statistics it's now called data science you guys know what a data scientist is? a data scientist this is a statistician who lived in San Francisco if you haven't heard this before I'm proud maybe it's because you're in New York it's the second time today I'm telling this in New York maybe the first time ever it's the same thing and it's a very very as you know it's a very important and very popular domain today and it rests on probability theory now what data science or statistics what they do for us is everything what we eat the technology we're using the drugs that get developed the things that get rejected or approved all of these things are based on statistics this is life and if we get it wrong and I just showed you how easy it is to get it wrong and we do get it wrong people lose their jobs and statistical associations have to remind people to read their frickin freshman books it's life so let's let's remember the words of the poet that was Sting questions I have a question about what you said in the beginning yes you mentioned a math school for only math yes could you tell me more about it yes I'll be brief a few quick corrections I did not mention it David did you're right you did it's not a school for only math it's a full curriculum school for kids who love math and it has plenty of math but if you're looking for a school that has only math I don't have one that school is in San Francisco it's called proof school and it's amazing so I encourage everyone who has school-age kids to explore it but it does teach also not math hi I have a question great lecture by the way I really enjoyed it I have a question about the very first puzzle that you gave us so you meet someone on the street they say they have two kids and they randomly bring up the fact that they have a daughter what's the probability that they have two daughters and you said it was one third it seems to me like it would still be one half and here's why if you ask her do you have at least one daughter then I agree it would be one third but if she's bringing up a child randomly in conversation and we believe that there's an equal probability a parent would mention either child it seems to me that a parent of two daughters would be twice as likely to mention a daughter it's a very good point and for that reason maybe I wasn't kind of being explicit but for that reason I try to recast the situation as a simple sampling from a fixed population so instead of the story about meeting a person on the street imagine you're literally looking at the entire corpus of families in the world of the US and you're removing all the families that don't have two kids and you're removing all the families with two kids that don't have at least one girl and you're remaining with a certain cohort of families and now you're sampling a random family from that cohort that's the right way to tell that story it's a more boring way to tell that story so I plead guilty but you're absolutely right because when you meet people you can never tell what why did she mention maybe we're in some part of the country where people only talk about their daughters so there could be a lot of confounding factors and that's why the it's another good point because there could be other subtleties here that you might not have noticed and you did notice but if you recast the question it's a question about a probability within a certain well defined population from which we are sampling uniformly at random then all those troubles go away and the answer is one third so guilty is charged thank you I have a question about your last comment about how statistics become this new field of data science and so given the prevalence of new tools to accomplish a lot of those data analysis tasks you know like R and just all these like statistical packages that we have like widely available often free to the user I was wondering if you think these are bringing us kind of closer to I guess a better approach to these to probability or if you think this is kind of making it easier for people to kind of whip up these you know statistical analyses in these computer programs and have no idea what they're doing which do you think this is really a benefit or I'm going to be very firm about this thank you for asking the question I think it's an important question maybe I should have said this like right away data science is an attractive domain today for many many young people it's talked about, it's lucrative it's great people flock to it look at Coursera, look at Udacity a number of ways you can study about it look at the number of statistical packages that exist out there to help you are was there for a long time R is almost basic now there are so many packages you can literally throw a data set at and they will run a deep learning model for you and give you back the result and what I believe this does to your question is get people away from spending time to learn the basics it's very attractive to skip that you can so quickly today join a Kaggle competition and do well just by learning the tools taking the data set picking a good library getting results and calling yourself a data scientist I don't think that's right I think that leads to situations like I think that leads to situations where you can get by but when anything strays by an epsilon from the beaten path, you're lost you don't know what's going on, you don't know why if people ask you to explain how the thing works you don't know I don't believe you can be a good practitioner of anything without understanding how it works fundamentally now there is a certain amount of like aircraft pilots don't have to know everything about aircraft engineering to be good pilots I get that but you have to know something I see this happening around me all the time people rush to the tools because they're there, they're easy to use and why should we bother with all those mathematicians who are annoying us with those different difficult problems with proofs, oh my god conditional problems I don't need that I'm doing deep learning, I'm way past that I'm not exaggerating this attitude exists I think it's dangerous and that's part of the reason why I'm speaking about those things so to your question it's the latter very much so Hi, that was an amazing talk it was really fascinating hearing about the paradoxes I have a question on the first paradox the same as the first question asker you said that the first parent you talked to had a child named Jane and I'm wondering how that's different than a child born on a Tuesday you caught me, I'd give this child again you guys are great a child named Jane is a lot less specific than in some ways it's a lot more specific I once again confess to letting my storytelling get ahead of my math precision I don't know exactly how to gauge the information latent in saying that the girl is named Jane but the right way to do this once again is to look at the cohorts of families ignoring names just look at the families that have two girls no matter what they're called in the first instance, all of them in the second instance, the ones that have at least one daughter born on a Tuesday you're absolutely right, if we know that one of them is called Jane that changes everything in the same way as Tuesday does and again, sorry, guilty as charged I'm trying to make this sound like a story and less like a bureau statistic sampling thing so again, guilty as charged hi, I want to go back to the 2008 research paper that you said helped invalidate the the hot hen 2018, exactly and I'm still bothered and I don't know if I'm the only one in this room but by the 44% yes to get ahead after ahead and I understand you say it's an empirical result no why 44% I'm lost great question and once again if I had more time so two confessions first of all, I don't have a good intuitive explanation for why this happens I want to be very clear about what it is that happens what we looked at there was a series of experiments all of them done with 10 coin flips and in each one we looked at the proportion of heads following heads their ratio is a fraction of anything following heads 3 out of 5, 2 out of 4, etc it was these numbers that's not an empirical observation, you can calculate mathematically we're talking about coins here not a hot hen this analysis establishes the baseline with which Tversky should have compared themselves against they assume that the baseline is 50% and it's not this can be proven mathematically with complete accuracy and I'm happy to stay around here for anyone who wants to explore further I do have a really nice story to tell that makes this a bit more intuitive I'm wondering if I can do this in 4 sentences right now so real quick I'm flipping a coin twice the chances I'm getting heads heads are what? 1 in 4 the chances I'm getting heads tails are what? 1 in 4 the same, right? again I'm not out to get you heads for 25% heads tails 25% now a slightly different experiment I actually do this with kids when I teach this split yourself in 2, you guys are gonna flip a coin repeatedly until you get heads heads like heads followed by heads and every time you do this you count how many times it took maybe 2 times, maybe 4 flips maybe 10 flips, right? you guys are doing the same thing flip flip flip until you get heads tails and remember, heads heads tails have the same probability you guys are flipping until you get heads heads you guys are flipping until you get heads tails and you're keeping count of how long this took I do this with kids and we write the results on the white board and you know what happens are the results the same? you know by now that they're not the same the average here is 6 flips the average here is 4 it's a big difference you see it why? if you don't see why come and ask me right after but there's a big difference so when we repeat experiments and we look at windows that are overlapping, something unusual happens and that's the reason that's the vague hand wave reason behind the 2018 paper but if you want to understand the 44% that's actually a calculation we have to do but it's not empirical, it's an actual theoretical calculation it's amazing thank you very much and let's give another hand to our wonderful speaker