 Okay, so hi, I'm Jessica Holman. I'm an assistant professor at the University of Washington. I study or research information visualization. And so my talk is called the visual uncertainty experience. And so when I proposed this talk, I thought it'd be cool to sort of spend all of my time talking about what I think is like the future of uncertainty visualization. New ways that designers are using and that I've been studying for showing uncertainty visually. But then I sat down and I started preparing my talk and I was like, ugh, if I'm gonna talk about uncertainty visualization at all, I have to talk about some not so exciting things, like probability distributions, variants. And so I was like, well, maybe I should retitle this talk so that people sort of know what they have coming. I could call it something like the trouble with uncertainty or maybe even more appropriately, I could call it stats 101 and people wouldn't know exactly what they were getting into. But then I thought, well, okay, maybe I can talk about these sort of fundamental aspects of uncertainty and the really exciting stuff and I'll just do it through a story. And so the story I wanna start with is out of Sarah Francis Galton. Galton was a Victorian statistician and Galton was thinking about, well, actually a lot of things, but one of those things was uncertainty. So Galton became interested in height, this physical property, and his interest was driven by a simple observation he made as he began to look around at the heights of people near him. He noticed that everyone has height, but we can't exactly predict what it will be. And this bugged Galton a lot because he wanted to know, how can I predict heights? I see heights all around me. So he actually starts measuring people's heights. So he's measuring families at first with children. He's trying to figure out sort of what is the set of possible heights? And so he starts to make observations from these measurements that he's taking. He has about 1,000 people that he measures. And so he figures out, for instance, that when you have parents that are unusually tall or small, the child's height is actually often closer to average. And this happens on both sides of average, whether the parents are very tall or whether they're very small. Similarly, the inverse is true. So if you have a child that's unusually tall or small, their parents tend to be closer to average and height. So what Galton had noticed here, we call regression to the mean. And he was really interested in this and he started to think, so what is this set of possible heights look like? And he kept making sort of more observations. So he looked at his measurements and he noticed that, well, the number of people with heights that are average plus or minus some amount, it decreases predictably and symmetrically as you move towards the extremes of height. And so he starts to see this vision in his head of what height looks like and it looks something like this. So here we just have the height scale and then to the right of that, you see this curve, looks like a bell curve. What he had discovered was actually in the normal distribution and you read it just in that as you move further right, that means there's a higher probability of having a height at that value. So Galton was really excited because he knew this was super powerful. So suddenly without any knowledge about the family's heights, other things that might help him predict, he could simply take the shape that he had and he could say, for instance, that given a child a random boy, his probability of being some height, say six one is 14%. So this was really, really powerful but he also knew that he could really, really confuse people with this because really what does this mean? So you have a random boy, you wanna know what's the probability he'll be six one, it's 14%. In actuality that boy is gonna either be six one or he's not. So what do we do with this 14%, what does that even mean? So he realized, well maybe if I communicate this as a frequency, people will get it because probability is kind of abstract and so he thought, well maybe I could say that like 100 boys, if we took 114 would be six one. So he decides, this is how I'm gonna communicate it and I wanna visually represent this. And so he draws a chart. So he draws his height scale and then next to that he has a bar and he decides to draw dots to show possible people. And so he draws a million dots and sort of the frequency of the dots in a given region means that's how many people there are with that height. And he annotates it. Actually he was a little bit wrong if you look at this. So height is not a perfect normal distribution. There are not as many sort of nine feet tall people as he predicted but that's okay. He basically discovered something really important. So what he had created was an uncertainty visualization. It's a distribution of possible heights that he's showing, he's representing it as frequency which ends up being important. People understand probability in terms of frequency much easier. So we could say from this example that uncertainty is simply the possibility that a quantity could take multiple values. And that makes uncertainty visualization the task of representing what those values are. So the interesting thing and I'm not the first person to say this is that most visualizations actually don't represent uncertainty. So people know, designers know and users know uncertainty is complicated. And so even though our data are always measurements they're always imperfect samples we often just don't show the uncertainty because we don't know what to do with it. And Galton also knew that uncertainty was complicated and so he'd done his best to visually represent it in his chart but really he was not very happy with this because he thought well if I wanna communicate this shape, this normal distribution I have to communicate it as a process. So it's a very abstract thing, any probability distribution but maybe people will understand what it is if I can show how it naturally arises from a bunch of events that are random basically. And so he makes something else to communicate this distribution which is more like a device. So he makes this thing out of wood and he has a funnel at the top and he drops little balls into it. And then with various probabilities the balls move through some pegboard and they sort of bin up at the bottom in these bins. And so you get back basically a normal distribution when you put a bunch of balls in it and it looks something like this if you were to play it out you're actually building the normal distribution and so Galton was way happier with this. He said okay now I can communicate with my colleagues that this normal distribution arises from sampling people from the population assigning heights from the distribution and you get back the same thing. So he was happy with what he had made. So I just wanna pause here and point out that just in Galton's work we have sort of a dichotomy, two different ways to show uncertainty. On the one hand we're showing all of the probability distribution at once in a single static graph. On the other hand we're showing it sort of one outcome or one observation at a time. We're building up the distribution at the bottom but the focus is on the individual samples. So let's move away from Galton fast forward to uncertainty visualization today. What's interesting is that when we do see uncertainty and visualizations today we tend to see these same two approaches one is way more common. So on the one hand we have these sort of static plots like error bars, violin plots to show probability distributions. We could also use something like a gradient plot map it to opacity. And I like to call these static aggregate distribution plots overall or sad plots. On the other hand we sometimes see occasionally not very often what I would call like sample based or experienceable uncertainty visualizations closer to Galton's board. We're actually showing people the data in such a way that there's a focus on one observation or one sample at a time. We can do these so that they build in sort of the depiction of the probability distribution overall but a very different technique. So one of the things that I wanna convince you of today is that most of the time if we see uncertainty visualization at all it is using one of these static aggregate distribution approaches. And I wanna convince you that there's problems with those. So let's take a closer look at them. So here's a bunch of these techniques on the very far left. We have things like error bars which are designed to show an interval in which some value can fall something like a mean height or just a height measurement. And next to that we have a box plot also showing through lines basically graphical annotations different properties of a distribution like the quartiles which are points that break it up into four equal sized bins. So I call these approaches sort of summary marks because we're using lines to summarize properties of a distribution. On the other hand we see things or other approaches that take the probability and map it to a visual variable. So things like area in this violin plot which is basically sort of just a PDF turned on its side and doubled so that it's symmetric. And then we see things like the gradient plot next to that using opacity to show probability. Okay, so what are the problems with these things? Well, let's start with these summary marks. So in infobiz research we have these criteria that we use to judge whether a visualization is gonna be successful. And one of them is which we can sort of summarize as expressiveness is basically does it express the data in such a way that the user's natural interpretations are accurate? So we don't wanna put something in front of the user that's gonna mislead them. And we see these summary mark based approaches sort of fail sometimes in this regard. So one thing that people do when they get error bars is do what or they show what's called within the bar bias. So they basically look at the error bar they see that half of it's on top of the data and so they think, oh, that part has to be more probable. Actually, this is not true. And so this is one problem. Another problem we have is that we know people don't like uncertainty, it's just complicated. And so when you show it as these lines on top of the data it's really easy for people to just sort of squint and say like, oh, I don't really wanna deal with that and it just kind of goes away. And they just, they don't use it at all which is a problem because we want people to incorporate the uncertainty into their judgments. Okay, so there's other problems that tend to affect the approaches that use visual variables to show the probability distribution. And so this other criteria that we use in visualization is called effectiveness. It's basically how easily can a person read the data values back from the visualization. And so we know that things like position are the most accurate channels. So scatter plots are great, length is also good. The problem is that often when we're visualizing on certainty we're already using those really good channels for other data to show the data itself for instance. And so we're left with these less effective channels like area which is hard for people, opacity also pretty hard for people. And so they basically can't read the probabilities back so that's not good. Okay, so these are what I would call visual problems with these static aggregate distribution plots. There's also what I would call model problems that deal with what we're showing people and how complicated and abstract that is. So recall uncertainty visualization is representing the possible values that a quantity can take. So things get complicated because what quantity we're talking about can differ. So on the one hand we could be talking about sort of the actual measurements. So what values can height take if I go out and I measure height from a bunch of people? On the other hand we could talk about a quantity that is actually a statistic that we calculate given a set of measurements. So what does the average height look like? What's that set of possible values? Two different things. Usually when we're looking at uncertainty visualizations things like error bars, we're looking at the ladder. So we're looking at the distribution of something like average height which is very different from the distribution of actual height measurements and it's called the sampling distribution of the mean. And it's basically related to the data distribution so this red SD is the standard deviation of that sampling distribution or standard error. We can look at the data distribution and the standard deviation divided by the square root of N which is the number of samples but nobody really understands this and they're not sort of broader audiences and even statisticians struggle with this. So people don't understand how sample size relates to variance and so this distribution you sort of tell them that's what they're seeing. I think maybe they think still that they're seeing the data distribution. You tell them it's the sampling distribution it's just like it does not compute at all. Okay so if I were gonna summarize these static aggregate distribution plots and their limitations that we've gone over I think I'd have to say that they are the folk music of uncertainty visualization and so I mean this in the most scientific way so I have some folk music for you to listen to so let's think about what this means. How many times can a man turn his head and pretend that he just doesn't see? Okay so why do I say this? They're the folk music of uncertainty this. For one folk music has complicated abstract lyrics lots of metaphors nobody really knows what the song is about they argue is it about politics is it about drugs we don't know and similarly static aggregate distribution plots things like error bars they represent something complicated abstract people don't really know what they're looking for or looking at. Folk music is also often unplugged so they could have used technology like amplifiers synthesizers they chose not to and similarly these static aggregate distribution plots don't make use of things like interactivity and animation which could help us sort of convey uncertainty to people. Finally folk music the lyrics often raise questions and then they don't answer them and I think similarly when we see things like error bars on visualizations we there are actually questions that we wanna answer and then we can't and it's really frustrating so I think this last point is really important so to me at least visualization is about comparisons so we wanna put the data in front of people in such a way that they can accurately compare the values and that's why in visualization research we care about graphical perception studies because they tell us what's the most accurate visual encoding it's position et cetera so we really care about comparisons and differences and the problem is that when we put a visualization like a bar chart with error bars in front of people they want to look at the differences and what it suggests that they should do is ask themselves are these differences reliable are they likely to repeat if I did this study again et cetera but they can't make that judgment so for instance I might have done an experiment I'm looking at blood pressure drugs or new drugs so I have my treatment I have my control the treatment group has lower blood pressure but obviously there's uncertainty because I have a limited number of people and so I want my readers when I show them a plot like this to be able to say sort of what's the probability that this effect would repeat again if I reran the study what's the probability even that I would see an effect of at least 10 percentage points nobody can answer that question easily and so we might think well people are rational right so they know they can't use the visualization to judge to make judgments of reliability and so they're smart they'll go look at other statistics you know significance testing which has problems of its own but that's actually not what people do so what they do is use the chart anyway because they can still make a judgment with it and so this has led Daniel Kahneman to say that when called upon to judge probability people actually judge something else and believe they've judged probability so this is problematic so what else are they judging if they see for instance error bars on a bar chart and they're trying to judge reliability so they might be judging for instance how much they like the authors of the study that's gonna tell them how reliable they think it is what I've seen when I've done studies to see how people interpret error bars is that they often look just at the difference in the means or the difference in bar heights regardless of the uncertainty so they'll say oh a big difference has to be reliable regardless and a small difference is always reliable so this does not always work but what they're doing is using what we call a heuristic it's a mental shortcut people use them a lot when thinking about uncertainty and probability they don't know how to do answer the question so they use some other information and heuristics help us reduce complexity we can answer questions more easily but we use them so often and they're so familiar that we end up actually being confident even though we're using these flawed sort of mental shortcuts and I think to people who are visualizing uncertainty this is really scary because what it means is that we can put a crappy uncertainty visualization in front of someone they don't know how to interpret it and yet they do anyway and they feel confident so that's not a good thing okay so what can help us so we see there's all these limitations of static aggregate distribution plots and sort of uncertainty in general so what's the answer so one thing that I've been thinking about which I think is sort of one path that could potentially help is to sort of step away from conventions of modeling uncertainty and how we tend to think about it and graph it in statistics and instead think about how we experience uncertainty and uncertain events in our everyday lives so for instance maybe I take a bus home from work every day and so every day I'm sort of waiting for the bus and I get a sense of when it's going to arrive and how that compares to its scheduled arrival times so maybe five out of 10 times I know the bus will be within one minute of scheduled time maybe another three out of 10 times it's gonna be two to three minutes late two out of 10 times it's really late, et cetera so I'm basically building a probability distribution in my head but it does not feel like that it's just a sense of expectation that I get from observing things over time and this has led Laplace, a mathematician to say that the theory of probabilities is basically just common sense reduced to calculus it makes one appreciate with exactness that which accurate minds feel with a sort of instinct so how can we draw on this when visualizing uncertainty how can we sort of show uncertainty in a way that kind of matches the way people are used to experiencing it so one thing that I showed briefly earlier is what I would call sort of an experienced based on certainty visualization where you're actually showing someone what the variation looks like so here's an example from the New York Times and they wanted to communicate to people how the jobs report which is sort of an interpretation of things like unemployment and job growth rates can be highly sensitive to variation in the months preceding whenever the report was written so what we care about is the overall annual job growth rates but we can write reports that are very sensitive to sort of what happened the last two months so to show this to the user they have all these headlines that could occur from the exact same annual rate depending on local variation and then at the bottom they have graphs so where we actually see on the far left sort of what we might expect if the job growth rate was actually steady but this is sort of hypothetical we would never see this this is a case where there's no random variation but then next to that we actually see okay so let's take this steady growth rate and show what it looks like with random variation so suddenly people get a sense of oh so I could have a steady growth rate and this is actually what I would see there's a lot of variation then next to that we're showing on the right side what we would see if it was actually accelerating but without random variation and then next to that what happens with random variation so people can suddenly sort of watch randomness play out in the actual data that they have and see okay these are other possible outcomes another thing I like about these types of visualizations is that we can as designers build in or build on people's natural associations with randomness so people do have experiences with randomness through things like coin flips and gambling and so we can sort of build this in and give them sort of a fun experience with uncertainty so this is from the New York Times also and they're actually using roulette wheels to show you in a way that you can play again and again what the outcomes could be for states for different states for the election finally I think these types of visualizations that are showing you sort of individual samples possible values are better at showing the process and in that way better at communicating what a probability distribution actually is so I think actually Nicky Case showed this one from Nathan Yao but we're basically seeing sort of the probability of living to the next year a bunch of different possible outcomes given some gender and some age and it's actually building up this little histogram as it goes so people sort of have a better intuition for what that histogram really stands for okay and then I think most importantly if we show uncertainty as sort of sets of possible values actual possible outcomes people can suddenly have a chance at answering questions about how reliable differences in the data are so we know this is hard with error bars but imagine that I'm showing sort of just a set of outcomes so suddenly someone can watch this over time and say oh I think about 70% of the time the treatment's gonna have a lower value than the control and they can even get more precise perhaps with some interactive help they could look for things like how often there's an effect of a given size or more another really cool thing is that you can show joint probabilities with these sample based plots you cannot show with the static aggregate plots so imagine in one possible world I've run my study of the blood pressure drug and I've done it as a between subject study so different people in control and treatment but then in another world I decided I wanted to run it as a within subject study so suddenly I have the same people in the control and the treatment they're doing both in randomized orders so what these sample plots will show that you can't see from other visualizations is when you have correlations between multiple variables so I can watch this thing and I notice oh when the control goes up the treatment tends to go up as well same vice versa and that's because it's the same person so there's correlation there and I can suddenly see that so what this is leading up to really is this thought I have that these experienceable uncertainty visualizations that show possible values really have to be sort of the disco funk of uncertain divas so let's think about what that means so I think you get the point so why do I say this well for one when you go to a discotech or a rave you kind of want to feel the music on a sensory level so you actually want your organs to vibrate and I think similarly some of these experienceable uncertainty visualizations are showing you uncertainty in a way that it becomes sort of a perceptual experience it's showing uncertainty as frequency over time which is something that we can relate to disco funk uses a lot of technology there's a lot of synthesizers you have DJs sort of mixing the same songs over and over with different sound effects and I think similarly these types of visualizations allow us to use things like animation and interactivity and I think we're just beginning really to explore sort of how we can use interactivity to set up kind of a conversation between the user and the uncertainty in the data finally anyone could get in to disco funk you don't have to know the history of that type of music to enjoy it and so there's this low barrier to entry and I think the same is true of these experienceable uncertainty visualizations so people can understand randomness without a statistics background so I just wanna spend my last sort of five minutes talking about how we could create these things so there's, cause there's sort of a general formula and I wanna show you really that you could do this for different types of data and for different types of visualizations so there's sort of three steps so first step you wanna generate from your data these hypothetical outcomes are hypothetical samples and the goal here is that we wanna actually generate these samples in such a way that they are representative so if we were to re-run our measurement process or our data collection process we could have gotten these other values so they're plausible and representative so how do we do that? Well, we could do that in a few ways so imagine we have our data set where we have our control and our treatment for this blood pressure study one thing we could do is simply look at what sort of distributions we have in our actual data and then sample from those distributions so maybe I find that my control and treatment are both normally distributed and they have a given mean so control has one mean and one standard deviation treatment has another and then I just sample over and over from distributions that are set to those parameters another way that I can do it even better because I don't have to assume what kind of distributions I have is to use bootstrapping or re-sampling with replacement from the original data set and Amelia actually talked about this a little as well and so nice thing about bootstrapping is that there's ways to do it for almost any type of data and there's a lot of literature out there there's books, there's stuff online so if you just Google it and what type of data you have you can find sort of guidance okay, so the second step is then to take all of these hypothetical samples we've made you wanna make a lot sort of like a thousand would be great and then visualize them so that they become separate frames in a visualization and to do this we wanna make sure that you can then compare the frames one to another pretty easily and so in most cases that's pretty easy in some cases it's trickier depending on what sort of visualizations you're using but sort of an example of why we need to do this would be to imagine that I have these bar charts where I'm showing the data using height of bar but I have different y-axis ranges on every bar chart that would be really hard for people to compare it's just like it's not very easy at all so we wanna make sure that those mappings are the same or are our functions that set the scales etc are the same finally we present them and you can do this in many ways and I think we're just starting to explore how to do it so you can do small multiples if you can shrink them down and people can still see the individual outcomes because you wanna show a lot you can use animation and I think there's a ton of possibilities for how to do the animation so you can do it sort of pure where you're only showing one sample at a time you could add the summary marks like the mean etc you could have it aggregate in to show the distribution you could have it be interactive so if you think it's too distracting to have the animation maybe people can interact and turn that on when they need it so I think there's so many possibilities here so the last example I wanna do is to show you that you can apply this type of technique to any data set really and so I thought I'd walk through this example from the New York Times so what they wanted to show is that we can use poll data to predict who's gonna be in the Republican debate so they did this before the debate happened but they wanna show how poll results are uncertain and so how we could actually run the same poll over and over and we might see different people predicted as debate candidates so they wanna basically show what happens when you have sampling errors cause polls are always based on smaller numbers of people than the actual population so there's error there so how do they do it so they're doing this all in JavaScript I just looked at their code and what they're doing is basically simulating many polls and showing different rankings that you get based on each poll so what they're gonna do is start at zero and they're gonna take the most recent poll results from the candidates the different Republican candidates so maybe in the recent poll Trump had 18% Bush had 14% et cetera and then starting at zero they're gonna create a bin for the percentage for each candidate and so for instance Trump would get zero to 18 then the next one we're just gonna build on top of that so Bush gets 18 to 32 Walker gets 32 to 43 et cetera so we do this for all the candidates and it should sum to 100 and then what we're gonna do is actually simulate each poll so we're gonna simulate a large number of polls maybe a thousand times and then for each poll we'll decide how big it's gonna be I think like 2,000 is sort of a common poll number so then for each of 2,000 people that we're gonna simulate we're just gonna draw a random number between zero and 100 and then put it in the corresponding bin for whatever candidate it falls in so the first one goes to Walker the second one goes to Trump et cetera so we do this for 2,000 people and then we've simulated a poll and we're doing this for many many polls and so then all we have to do really is for each poll after we've sort of put all the people in the bins we calculate the new percentage for each candidate we sort those and assign rank sometimes the ranks are gonna change so people are gonna swap so we're actually gonna predict different people are gonna be in the debate when we do this and then the last step is simply to visualize it so we can use something like D3 visualize or create a function that visualizes a set of ranked candidates in order to make it easier to compare we're gonna animate transitions so that people don't have to keep track of where each candidate was they can sort of follow just from the transition okay so I think I'm about out of time so thanks for your attention if you're interested in more on this technique of experienceable uncertainty visualizations you can check out my website I've also written some stuff on the interactive data lab blog on Medium so thanks I think we have time for maybe one question if there is any so you can study it in the same way that you study the other stuff so people are just beginning to do stuff on like looking at how people average a bunch of dots over time so we don't totally know so obviously it's different and in some cases it might be less precise but I think that's one of the goals is sort of understanding how to design these things so that it actually works so basically it's an open question there are a few studies that I could point you to but we don't totally know how it compares yet so awesome thank you so much