 Thanks. Near the end of his presidency, one of Thomas Jefferson's friends wrote him a letter. And he said, hey, I'm paraphrasing. Hey, I'm thinking about becoming a newspaper publisher. Could you have any advice for me? And Jefferson was pretty skeptical of newspapers at the time. And he said, maybe one thing you could do is you could do your thing in four sections. The first one would be truth. The second one, probabilities. The third one, possibilities. And the fourth section would be lies. And he said, the first section, truth, would be very short. And so today, what I want to talk about is how we have room for the second section. The newspaper that I work with hasn't followed this model. We use things like travel and sports and national to divide our content. But we do sometimes say explicitly that we don't know. It's become part of our playbook for really terrible things that we have a section. Here's what we know about something that happened. And here's what we don't know. It's sort of interesting to me that we almost exclusively do this for terrorist attacks and bombings and fires. We do it sometimes in the database world, two years of the map of the 1916 election, which is pretty incredible that they could pull it off in the amount of time they did. But you'll see that there's a doubtful category. Abbreviated DFL, I think we should bring that label back. I think it would be fun. And it turns out in this instance, the doubtful category wasn't quite wide enough. There are at least three mistakes on this one, like Minnesota and I think Idaho and New Hampshire, they're the wrong color. So doubtful, they should have used it a little bit more. But this isn't really the type of not knowing that I'm so interested in. I'm a little bit less interested when it's black and white than when it's like a little bit more murky or a little bit more fuzzy. And so there are little real examples of people doing that. When I think of murky and fuzzy saying, I don't know, I think of these guys. And these are the sea monsters that people used to draw on old maps, maps in the 1500s and the 1600s. And like database, people were trying to pull off a lot of things with sea monsters. Sometimes they were just trying to show that they were fancy, they were kind of expensive, you had to hire a special guy to do it. Sometimes they were just trying to fill some space. But my favorite ones, the ones I like the most, are the ones where they were trying to indicate that there's something here and I don't know it and it might be dangerous. And so there's something in between, they're saying something weird happens there, I don't know. Some of you can probably guess the reason that I've been thinking about uncertainty and how we say things that we don't know. And that's this guy, which was an election night offering. And this is how the upshot interpreted results on election night. So there's a few things going on. We've got three main numbers up top. We've got the chance that we think Clinton wins the presidency and then the popular vote margin, what that will be and then what the electoral votes will be. We've got shaded bands. Those are essentially like 95% sort of confidence interval-ish type things. And then we've got this needle that moves back and forth and it stays in the middle 50% of the simulations. And it's using Perlin noise, which is the type of noise that's been used to make marble and clouds and fire. And Gregor Eisch, my colleague, implemented that. And I think it's often graceful, but many people disagree. And so there's like two camps of people. There's people who feel like the needle helps you for the first time, understand it, to really feel the uncertainty in a way that you don't, if you just show a shaded band. I think a lot of data of his problems are about hierarchy and the needle moving forces you in some ways to not be able to ignore it, to ignore that these are estimates that were uncertain about them, that we don't feel more confident that any one particular number is true than the others. And there's other people who feel deeply uncomfortable by a moving needle who feel it's not helpful for them in understanding what is confusing about the world. And so still, there's lots of things going on on this page. There's the state estimates, the same sort of game, and they shrink over time. And then we're also keeping track about how our estimates changed over time in a couple of different ways. One, just summarize the uncertainty. And two, we show electoral votes with some bands. And I'm interested now in that when people, when they are, I think, angry on Twitter sometimes, they just, they tweet this image to say, like, who are you to say anything? But I read it as like, who are you to update your expectations in the face of new information? You know, 30 minutes faster than the betting markets and three hours faster than the networks. But I think there's too many characters for Twitter. So they just, they just use the image. And no one ever uses this one. And I think it's because this one is only understood by people who are like, don't need to be aggressive on Twitter. Because just as, and so, lots of the work that we've done, I accept the fact that people interpret it in different ways, right? And of course that has to be true, that people take their whole life experience to your work and their knowledge about things and how everything that encompasses their life. But they've never felt more than with this graphic about how people feel things differently, how, you know, it doesn't feel the same to everyone. And that shouldn't be surprising to me at all. I think of this guy, which is a chart experiment where someone asked a couple dozen NATO officers, tell me what you think about these words. Tell me what you think when I say probable. So it is probable that. Give me a number about what that means. Or it is, there is little chance that. Give me a number about what people think. And it's stunning in some ways that, you know, these are NATO officers, they're all English speaking as their first language. And probably, if I said probably, some of you interpret it like 40% and some of you say 90%. So anytime we're ever trying to communicate anything, whether it's through pictures or through words, there's this like gap between what you say or what you think you are saying and what people hear. And so this is older, they're special, someone on the internet, someone on Reddit, redid it with just people on the internet. And here, you know, you get like people who are a little sillier on the internet that highly likely does it. But the shocking thing is that, it's like basically you get the same answer. Whenever we say anything, you know, there's this wide interpretation of any word that we use of any image that we use. And so I think this, oh, this is fun. Because in some ways, when I think about the Trump-Quinton odds or some work that says maybe, it would be better if you always present those in terms of the adverse outcome. So we could, you know, figure out whether our readers think an adverse outcome is Trump or Clinton and then present it to them in that way, right? We could just like personalize it. Or we could figure out whether you are the sort of person who thinks highly likely means 70% and just like up that a little bit in our verbal descriptions of them. Of course that's not gonna work because thinking it turns out is different than feeling. And we know that from the psychology experiments where people try to figure out how do we feel probability? That I know in my, I can know in my head that 50%, it's just 50%, it's like a coin flip. But I don't feel it in my heart because I tend to overestimate the small probabilities and underestimate the big ones. And it's also possible that I wanna be lied to some of the time. Here's one of my favorite chapters from Nate Silver's book where he talks about weather forecasts. And this is the National Weather Service where the, when they say 20%, they do mean 20%. That's what it means for the gray dots to follow the black line. This is a weather channel. It's possible that they're cheating a little bit on the low end, which you could maybe make sense if I don't bring my umbrella and it rains, I'm gonna be madder than if I do and I just have to carry it around all day. This is a local TV stations in Kansas, let's somebody get it. And so I'm pretty sure that I don't want that, but I'm not certain that I want the one that's actually perfectly true. It's possible that in some contexts I want to actually be lied to. And I think that that's the answer about why some of this is so difficult and why we're never gonna have a blanket answer. It's because it depends on context and it depends deeply on the norms of a discipline. No one was upset when we played with a little jitter on primary night, at least that I heard about. But the jitter on election night upset some people. In the same way, we think of this as a normal way to represent uncertainty when we discover earth-sized planets, not when NASA discovers them. And in the times how we signal that this is not quite real, we call it a rendering and we use words like some of them could have surface water. But there's this beautiful article in the Atlantic that explains what they are looking at when they develop these things. And so the first picture that the rendering comes off of this data blows my mind in some ways in the most delightful ways that this is the data that we have and if you know how to read it, it has to be consistent with certain types of things. The thing that the rendering loses though is that it loses all of the conversations and the arguments they have about like, does that green indicate that there's too much like that there could be plants there in a way that we don't want to? So the thing I'm interested in is what are the forms that we use to say there could be plants there, but maybe not. I don't know. The classical one is this one. This is a government chart. Tomorrow I think will be an important day for the government in federal revenue day, maybe, who knows. And this is the chart that they release every time the federal government releases a budget. And I love it. I think it's one of the most delightful things, in part because if you know how to read the scale, this is like the depth of the Great Recession and this is like better than it's ever been before. So essentially this chart says like, I am pretty confident that it's not gonna be like the Great Depression, right? Like five years from now. It's not even my favorite government uncertainty chart though, that's probably this one, which comes from the economic report of the president in 2016. This is about how low the unemployment rate can go before it sparks inflation. And my favorite part, I mean, I think your laughter suggests you'd get it, but the footnote at the bottom, it says a 50% band is used because increasingly higher levels of confidence produce confidence bands that approach unboundedness, right? And so it's like, it's even better than it even appears in essentially that it says like, I have no idea what this thing is anymore. But that's the point. That's why they use, that's why they use some sort of an uncertainty band. And I'm kind of convinced that that's our norm, that we only do it when we're forced to. In the archives of the times print graphics since roughly the last 25 years, I think there's about somewhere on the order of 4,000 times when we have said either estimate or forecast or prediction, like some word in that family. And I can find only eight when we've formally expressed some type of confidence interval. And I think most of them are when like, we felt like we had to, we felt like we were forced into a corner like this one where we wanted to say that childhood obesity rate is not rising, but if you just showed the like point estimates, it was like, well, kind of this. And so I'm basically convinced that we only do that. We only show it in a chart in some way when we're forced to, when we want to conclude the opposite. There's one small exception in that in that I'm not counting the times that we say like a margin of sampling error in a pole graphic because I am convinced in my heart that that actually probably does more harm than good. Like I'm not, I don't want to claim to you today that like introducing more uncertainty into the world and a little bit more jitter into your life is universally a good thing. And the example that I would claim for that is this guy. One of my proudest moments of upshot stuff is when we pull off like data stunts. And this is an example of a data stunt where we conducted a pole in Florida and then we gave the data to the raw data, the individual responses to four other people. And then we just said, tell me what I should have said. Tell me what the answer is in the top line number. But if you're just gonna not read any cross tabs or go deep about what this means, just like give me the answer of the pole. And it was more delightful than I even dare to dream because we get four of them and we get four answers back, right? Like so ranging from Trump plus one in Florida, which ended up being the final result to Clinton plus four. And so this is not accounting for sampling error at all. This is not, because we're all working off the same data. It's all exactly the same data. And so this error, this difference is about analysis and about how you actually read the data is totally on top of the normal margin of sampling error. And so I think it is indicated of sometimes problems in data this too, where we solve the problem that we think we can, right? Like we know mathematically, we can write down what sampling error is. This kind of stuff, you have no idea. Like you don't know what the turn out error. And so we focus all the time on sampling error and we put it in our footnotes and we think it's enough. And it turns out it's not really enough. In the same way it happens when you just plot the data that you have because it's the data that you have and what else am I supposed to do? And I have a closed form or a not closed form formula for it. This is what we do. There are other examples in the Times archive where we talk about uncertainty but we just don't use the word. So like when I search the archive, I don't find them. The classic one, of course, is a hurricane chart of some sort where our style now, and I didn't know this before this talk is like, there's no indication of what that cone is at all. It's just like, it's the norm for like hurricane. It's like a hurricane cone. You know, it's like, and until people started talking about uncertainty recently something I didn't realize is that's like a two thirds of the cone is made by like going out points so like six hours, 12 hours, 24 hours and figuring out empirically how off have forecast been for the last five years. And so there's something delightful about that is that the cones are always the same size no matter what they're different if they're in the Atlantic or the Pacific. And there's some part of it that I think is fun that pretend you got omniscient. Pretend that this was prophecy instead of probability. Your hurricane pass would be perfect and the cones would be increasingly wrong for the next five years. Assuming that you didn't also change the way we play the hurricane game but if you're like omniscient then you probably have better things to do anyway. But so we are comfortable with this kind of idea with this kind of a cone but I think it feels totally different sometimes when you see the actual path of the hurricane. This is the same hurricane that we were looking at in that map and if you watch it, it's like a double loop, right? Which is totally crazy. My favorite, the Twitter comment, the first one on this is like utterly ridiculous yet somehow not out of the realm of possibility. And I think that is a feeling in data-vis that we can sometimes lose if when you just reduce it to a cone and you just reduce it to a hurricane it's like gray and soft and in the back and says like, please don't worry about me except like maybe worry about me if you really care, right? This graphic, like implied sort of craziness of it, the model run for the same hurricane Matthew like is, it makes me feel totally different than the cone. And in part, it's just, it's design decisions and it's decisions about aggregating and it's decisions about how soft the weirdness is. And so what are the design decisions, other design decisions that affect that? I think of something like this guy, this is the only time I think in my New York Times history when we've used like XKCD style axes, like squiggly axes like the Randall Monroe style. And I'm not, and there are a certain set of people that think like sketchiness is the answer to uncertainty in data-vis and if we just started like drawing in crayon it would be like clear immediately. I'm not, I'm not certain I agree with those people but there is something about form. I think a lot about as recline when he talks about on his podcast how it feels like a looser medium. It feels like a place where he can try on ideas and it's clear even without him having to say it that he's not certain that what he's saying all the time is true that like there's some probability that some of the stuff is true. And so this is an example of a chart where we asked people to draw it. We've done a few of these now. This is one of the first ones and we said, so the X axis on this chart is your parents income percentile and it's as a rank. So there's like a hundred equal bins and the Y axis is the chance that you go to college not like a fancy Massachusetts college just like any college, any college at all like a trade school anything just and not even graduate just in role at some point. And so I thought maybe it might look something like this. Maybe it's an S shape that like once your parents make $300,000 like it doesn't really matter that much anymore except maybe there's like maybe a little trust fund dip here like at the end, right? So maybe it looks something like that. But the truth is actually the truth is like a super, super, super straight line and the straightness of the line I actually think of as shocking in terms of what it says about class in America that like you go up one percentage point or one income rank and that changes your chances of going to college by the same amount about no matter where you are on income distribution. So it's exactly the same for poor kids and rich kids. And sometimes I like to give this, talk about this example in talks. I was reminded yesterday when I saw Noah Veltman who will talk this afternoon about his reaction to this that like it may be inequality in America looks like the Frasier skyline or Eric Hinton who's like I'm pretty sure you just drew the elephant from the little prints. But the reason I think I like it for when we think about uncertainty is when I talk about it in certain forums, people who really understand data are confused about the fact that the gray dots are real data. You think of it as model. They're like too perfect. There is something in us that can understand that like that's not what real data is supposed to look like. There's something broken about its straightness. The other reason I think I'm interested in it today is I do think that our avoidance of uncertainty has real policy consequences in the world. So for example right now, the federal government, one of its best tools for thinking about whether this chart is a problem or not is Pell Grants. But Pell Grants happened super late in college. And so maybe by the time you understand that you qualify to get a Pell Grant, it's like too late for you to do anything about your high school transcripts or to prepare different. And so there's researchers who say, but we could do a pretty good job, a very good job actually of predicting who needs a Pell Grant by just looking at who qualifies for a free lunch in the eighth grade. And it's like very good enough. But there is some uncertainty to it. There's like 10% or something of kids who like, we would say, if we went with this policy in the eighth grade, we would say congratulations. We will give you a Pell Grant to go to college. So now you should start to think about your high school career a little bit differently. Who wouldn't by the time they actually enrolled in college end up qualifying. And there's lots of reasons as a country that we don't think of this as good policy. But I think at least a little, a small one is because we're uncomfortable with being a little bit wrong, even if it makes everything else so much better. So when I think about something like the election chart, the reason that we can be faster than the networks is we're saying, I'm comfortable with this uncertainty. I'm comfortable saying that I might have to turn around at some point. That if I say there's a 95% chance that this is gonna happen one of the 20 times I should be wrong, assuming that I'm not calibrated like the weather channel. I'm not cheating in some way to lie to you because it's better. But I wanna close, and so I do think that if we got more uncomfortable with uncertainty, if we got more uncomfortable with the fact that we don't know the future, but that we can have educated guesses about things, I think that there are real world implications and policy consequences to that. So I wanna close with a little bit of at least like a hopeful note. In that two months ago, the Federal Reserve started like producing these things called fan charts when they say like, here's what I think the interest rates are gonna look like. And the fun part about the story for me is that the Federal releases their minutes of their meetings with a five year leg. And so you can go in and you can see that 10 years ago, Janet Yellen, who's now the most powerful woman in America, said like, hey, I think we should put some uncertainty on our charts and you can see how important it was. It was like page 255 of 255. And you can also see that like, they went with 70% intervals. And so by the end of 2008, they thought the federal funds rate would be something like this somewhere in between like four and 6%. If you remember that time, it was actually like zero by that time. But you know that, you know in a 70% interval, it should be outside of it some of the time. And so my hope is that you believe that if the most powerful woman in America is willing to fight for this for 10 years, that maybe some of us should too. Thanks. And my graphics is the work. Yeah, so the question for those of you who can't hear is how do you talk to your colleagues, especially those who are more focused on words about when you put more uncertainty either into words or into graphics. And there's, I think there's this tricky balance a lot of the time because you don't wanna end up saying nothing, right? Like there's this class of words that I think of as weasel words which are like could or maybe or perhaps. And you can like, if you just stick like a perhaps in something, like the amount of things that you can say like opens way up, right? Like in a helpful way, not in a way with a ton of rigor. And so I think lots of the balance, especially in words, is that you wanna stay in the area where you can say something that is true. Like you want the Jeffersonian like first section, right? Like it's, and I don't wanna like, but that takes a bunch of stuff off the table. There's a thought in journalism that lots of journalists are like far more interesting than what you see in their stories. And part of the reason I think that is true is because you want the things in your stories to be 100% true. And then like if we were at the bar tonight, I would be comfortable being like, it's like the Ezra Klein podcast. Like I'm gonna back down from that a little bit. I can say it and it'll be like 97% true. And so especially with words, I think we don't have a ton of tools to treat uncertainty with rigor. The guy who made the NATO chart or who like was behind some of that research used to fight about how there were like poets and mathematicians and like the poets and the writing would always screw things up. And that's why we couldn't have nice things. But I think it's a balance about like wanting to stay, wanting to stay true, but also wanting to say something. And that's the inherent tension. Yeah, so the question is when you show some sort of probability estimate to a general audience, do you have to explain it? Or can you just like throw it on there? Right? Yeah. And I think that's a really tricky question. There have been examples where part of looking at the archives about how infrequently we do explain it in a formal way made me sad. And then it was like, how on earth am I giving people a chance to understand what we do when we do it if you only see it like eight times in 25 years, right? Like it's not like a repeated exposure example. But on the other hand, there are times when there are stories about especially certain types of statisticians becoming like Bayesians when they try to explain a confidence interval to people because they're just like, I can't do it. And I feel that in my own heart too that like occasionally I will be sent paragraphs about like people trying to explain what a p-value is. I'm just like, just don't do it, please. And so like I think there is a sense like your intuition is right, even if it's not like 100% technically accurate about like I get a sense that I don't know, right? For example, I'm not, I've never been bothered by the hurricane cones. I just trust that that's like a hurricane cone, right? But at the same time it like blew my mind when I understood those were just empirical, right? Like that they shift every five years and that they're the same every single hurricane. I'm like, that's crazy. So I think there's levels of understanding and you can understand it in a sort of intuitive sort of I kind of get it with without needing to know some formal definition about what it is. The other thing, the other trick is that usually the way these things are where they come from that comes with like a whole another set of assumptions, right? Like so the hurricane ones are clean because it's just like we took the last five years of hurricanes and we checked how many were outside this band in the last 12 hours out. Most of the time if it's coming from a model or something more complicated, there's this entire other set of assumptions on it that like you take to be true and then this is true. So that's an extra level of like complication in the labeling. Los Angeles University, thank you very much for the talk and your question and your work. I have two questions. The first one is that I love the example of the needles for explaining my students uncertainty. So I saw that you had like the whole rerun of the thing in there. Is there any way we can access that? That's a question. The second one is giving the response to the piece like now will you do it in a different way or will you stick to the web edition? Yeah, so I'm happy to share with you the rerun and that's like the one I made for today. It's out like, I think it's at 120 speeds. So the election plays, you know, that night plays in two minutes or something but I left the needle jittering like at the same rate. So it doesn't feel like frantic. Like I think people when they close their eyes and think about it, it feels far more frantic than it actually was in real life. It was soft and gentle, you know. And the second question about what will we do in the future and I don't know the answer. There's some probability, right? And so if I say like probably and some of you interpret that as 40% and some of you interpret that as like 90%, it would be fair. Yeah, I think, you know, obviously this was teamwork so a lot of credit goes to Greg Orish and Kevin Qualey and Nick Cohen and Josh Katz and lots of people and in our conversations about this, we wanted to help you feel the answer, right? Like I don't think we thought it would be as visceral as it actually is but I think visceral data this is good, right? Like I think things that you can like remember and feel and I don't know and I'm still uncertain about what uncertainty is supposed to feel like, right? Like my colleague Kevin Qualey argues that like this was mostly just raised people's blood pressure, right? Like does, is raising blood pressure what uncertainty is supposed to feel like? I don't know. But I do think it gives you the answer in a way that the maps don't. Like, you know, there's a point at the night where there were like a top level other in my office being like, why is this so different than the map? And it's just like, because you don't know how to read the map, right? Like Wisconsin being pink suggests the same thing the needle suggests right now. And so I think good, I think there is a good, you know, there's this trade-off about how like intellectual versus visceral do you want to be? And I think you see that with a lot of the confidence intervals that like those are intellectual things. I have to stop and think about it. The needle, and they were there too. Like, and I actually liked the way they like sort of shrink over the night, but the needle, I think you feel it in a better way. So I don't think that we are actually like, our goal is not to change policy. I think our goal is to inform people. And but I do think that's related to policy. Like in the in the Pell Ground example, you can say like, this will help 90% of kids and 10% of kids, we will just like make a mistake on it, right? And those, you know, like, not a mistake. Like we gave Pell Grounds to kids who are like slightly more middle class and we should have like, you know, horror of all horrors. You know, like, and so the idea that, I think it's really just more of a general idea that that sort of things that we are comfortable with is different depending on how complicated they are or how much we can, how we can understand them. I do think the question was if you, if we ever did the needle again, what range would it be bounded in, in its driftiness and would it, should it be more? I agree that I think, I think we are too conservative in bounding that to the, to the, you know, the 25th to 75th percentile in that I think it would have been better if at the beginning of the night, it drifted into Trump range, right? In the sense that like those, that outcome was like perfectly consistent with all we think we knew, but people didn't feel that way. And so I think at the expense of thinking, of people thinking that the needle is even like crazier than they already think it is, like that's cause it's true, right? Like that's cause that is within what we do not know. And so, you know, those are arbitrary numbers, but I think, I think if I were to ever do it again, I would push for more drifting. Have you seen audiences attitudes towards forecasts and uncertainty change over the last year? I think yes, I think they are more skeptical than they were in the past. I think there's a giant group of people who thought Nate Silver was a God. And it turns out he's a very smart man. And so how, you know, that the betrayal that people feel about that, even though, you know, he did, and I think he speaks eloquently about like, you know, 30% means 30%, right? And that's not because we don't have needles in our hearts or something, we don't feel that way. And so I think there is a general, there's a greater skepticism than there was a year ago for better or worse, maybe, you know, I think it's possible that expectations are just calibrated more properly than they were in the past. How do you A-B test for data literacy and how do you know when you want to go out on a limb? We don't do a ton of A-B testing within individual pieces. So I think it's more like, as a body of work, you know, you know, I know that there are times in my career when I've like sort of walked off the edge, but I also feel that like you have to like find the edge. So you know how not to walk off of it, right? And so we don't, like, because a lot of the cycles that we work on are pretty quick or we're just like bad at being disciplined, like we don't do a ton of like A-B testing within individual graphics in part because it would be weird in the social world, right? Where you're like, we don't control the distribution of these things in the sense that like lots of traffic comes from like Facebook or Twitter. And so it's really frustrating to like see the screenshot and then like not be able to like achieve that screenshot or whatever, right? That's solvable probably. So the, how do you know when you've gone too far? I think it's like when you feel like you can try to mediate that with clear language and clear labels, I've sort of, you know, as I've grown up, I've come to think that like when you need the like how to read a chart, like that's probably when we've gone too far, right? Like if I have to have the key that says like explicitly, here's your instruction manual and probably in trouble, but there are times when you feel passionately about that. So it's just a, an unregressed balance about what is far enough. How much time do we have? Maybe one more. What's the difference or how do you feel about simulations versus confidence intervals? I think I tried to answer this a little bit. I think simulations feel more visceral in a way that confidence intervals are good for my head and simulations are better for my heart. I think how I feel about that, but that's a, that's an idea in progress. You know, like the frustrating thing about simulations is then it's like, well, I want to see the summary, right? Like, and so I think in the dials, we're trying to do both at that same time. We're trying to show you like, you know, here's one that you can ignore if you want and here's one that, but if you want to like think about and try to remember the history of these simulations without trying to like rewind in your head or, you know, stare at it for seven minutes and hope you get the impression of where, where it was. I think both at the same time can be effective. Thank you.