 The second NCAR seminar for this week, MQ seminar for this week is about to begin, thank you. Tonight's speaker is not even night, but it sounds more formal that way, is Rebecca Morse. She got her BA actually from the University of Chicago in chemistry of all silly things. And then went on to MIT where she did her PhD with Kerry Emanuel on predictability and things of that nature. While she first came to MQ as part of the advanced study program in 2001 and became a scientist one in 2002 and has been working her way up to a scientist three from 2009 to the present. During that phase she seemed to leave the mathematical or pure sciences and went into this low level social science stuff. I'm trying to get fired from this job. And actually today she's going to talk about understanding communication interpretation and the use, wait shouldn't that read the mo-cast? Weather forecast and warnings and I just stopped there so I don't screw up any further. Thanks Morse. And I will admit I did tell Morse he could say whatever he wanted so remember not to do that. Yeah, I know. Remember not to do that if you're giving a talk. Okay, so before I start I wanted to thank my collaborators on the projects I'll be talking about today. So on the left side are the collaborators on most of the projects I'll be talking about. Their names are listed on the bottom of the slides. And then on the right hand side are the collaborators on the last thing I'll be talking about which is a new grant that I have from NSF. And of course there's many other people that have contributed to this work. And then also we have funding from the National Science Foundation from several different grants and from NOAA. Okay, so an overview of my talk first I'll provide some motivation and background for how I got to be doing what I'm doing and why it's important. Then most of my presentation I'll talk about research to understand and improve creation and communication of weather forecasts and warnings for protective decision-making. So I do research in a lot of different areas but today I decided to focus in on hazardous weather information for protective decision-making. And I'll talk about two different kinds of weather flash floods and hurricanes. I'll just talk about the flash floods briefly because I thought it would be interesting for people here since we experienced a flash flood and the study was in Boulder. And then most of my talk I'll talk about a few different projects on hurricanes. And then I'll summarize and talk about some directions for future work. Okay, so people often ask me how it is that I came to do what I do given that I have a PhD in traditional atmospheric science. So when I was working on my PhD the goal of my PhD project it was on adaptive or targeted observations and the goal was to identify locations to take additional observations where you would maximally improve the forecast to increase societal value. And so that was the goal and when we started working on the project of course we realized there's a lot of things going on that limit the ability for observations to improve forecast so really what a lot became about was how to work with the data simulation system and numerical modeling and predictability limits to identify places to take observations given that that would improve forecast. And we also the whole goal of the project was really to maximize societal value and we kind of thought that somehow that would feed back into the project but we never really got to do that. And so we just were making these sort of assumptions saying that if we improve the forecast in this way that will benefit society. So I was interested in looking at that aspect of the problem and so when I came here as a postdoc I started to try to look at the other side of this question. And so some of the first work I did when I came here was really trying to look at the whole system with all of these different things going on. So communication, dissemination, interpretation of information, how people use the information that's provided by forecasters in conjunction with other information and a lot of other factors are involved and how the different kinds of people that create and communicate and use information are interacting with each other. For example, public officials and members of the media and forecasters and members of the public and so on. And so I did some work looking at the whole system from a policy perspective and also from an economic perspective. And I had one project where I tried to take observations and trace them all the way through the forecast system to societal outcomes and it really became a lot to manage all at once. In large part because we just didn't have that much knowledge on the right hand side about how weather forecasts are used in these situations. I did some research over the years in this area, but it really wasn't very well developed. So I decided that in order to look at the whole problem and understand what was going on, it was really important to focus on that right hand side. So most of what I'm going to talk about today is really in that part between the forecast information, the societal outcomes. And what I do is informed by the knowledge about how the forecasting system works based on my role here at NCAR and a lot of other things and my background as an atmospheric scientist. So the projects we do, most of them incorporate all the things on the right hand side. Today I'll focus on use and decision making, how people use information and decision making in conjunction with other information and other factors. But I'm only talking about pieces of the project, so we do a lot of work on how different kinds of users interact and all kinds of things like that. And then when I get to the end, what I'll talk about is a new project I have. We're really trying to tie the whole thing back together. So now that we have more knowledge about the communication and decision making and so on, really looking at the whole system altogether. I think that's the future. Okay, so just to motivate this a little bit, this is some examples of the messages that were out as Hurricane Sandy or Superstorm Sandy approached the coast. You can see here the classic Conan uncertainty graphic for the National Hurricane Center issues, that type of version with a track line and without the track line. So this is the kind of information that meteorologists produce. You can see some other examples. These are all messages that I pulled off the internet and the media as the storm was approaching. So you can see this is an example on the left of the kind of message that the National Hurricane Center might produce. They talk about kind of in a science way about the risk of the storm. And then in the media, sometimes there's some overhype or kind of talking about it in this more dramatic way, like the perfect storm or Frankenstorm or Superstorm. Then you have public officials that issue evacuation orders and make recommendations. And you can see some examples here during Sandy. At the state level and even the national level, there were recommendations to evacuate. And then you have people that get all this information about what to do. So a lot of people evacuate. We know that and they protect themselves. But some people in areas of high risk don't evacuate. And so this is an example of just one person who before the storm was saying, oh, there are people who panic and evacuate and there are people who've been here a long time and they're unfazed by it. So this is kind of the kind of thing we look at and given all of these kinds of messages, how do people make decisions? How do they think about what to do and how do they interpret their risks? So this is an idea that kind of the trend that is happening in meteorology, particularly in the National Weather Service. This is the National Weather Service is relatively new impact-based tornado warnings. So these are tags that they can issue with tornado warnings when they think something is going to be very damaging or catastrophic. They also have this base one that I didn't show. So all caps are because this is how the National Weather Service issues their statements. I'm not trying to shout at you. You can see the language here that they talk about in the situation, you know, expect trees to be uprooted or snapped and then in the catastrophic damage they're really trying to say, this is going to be a really bad hurricane, you need to do something. And so there's a lot of debate in the community about whether this should be done and how it should be done and how effective it is and so on. There have been cases recently, for example, in a tornado a couple years ago where a lot of people thought that they were going to die and they got in their cars a little bit later in the hurricane context. Okay, so just to give you an idea of kind of what's going on in the U.S. and internationally, these are a couple examples of reports and plans that have been developed over the last 10 years talking about this kind of thing. So looking back to 2006, there was a National Research Council report that talked about the importance of understanding user needs and effectively communicating and this brought in social science. In fact, I was on this panel and at the time really our expertise in the use of information was in the climate realm and so there really wasn't as much work going on in weather there, but if you look forward in time it's really come into the weather community. So the National Weather Service Strategic Plan, they had this kind of stuff all over their plan. They talked about making our information more relevant to decision makers, linking social and physical sciences to produce and communicate information and then the last one is at the international level this high impact weather program that I've been involved in where one of their five themes is on communication and really to improve decision making communication of forecasts and warnings. So this is giving you the idea that really this is a growing trend in the atmospheric science community a lot of people are talking about it and trying to do it although a lot of people really don't know quite how to do it yet but there's a lot of interest. And you might think that census has been talked about for all this time that a lot of the problems have been solved but they haven't been. Even looking back to the NRC panel that I talked about the Weather Service funded that and communicating uncertainty and so fine so they weren't all this bad but this is from the east coast north from a few weeks ago in Boston got a lot of snow and New York got very little and then National Weather Service was apologizing in the media this is Louis Utolini who's the director of the National Weather Service the storm had been forecast to be historic and in some places it wasn't and there was a lot of backlash a lot of media discussion about communication of uncertainty and why this wasn't communicated well and so on so the problem is to be solved and it's kind of in the national media that's really affecting meteorology and the Weather Service and so on. So the kind of thing that I do is really motivated by atmospheric science research so my background is an atmospheric scientist and also kind of the problems I know that are going on in the research and also by challenges in communication practice like the ones I just talked about is to conduct research to help build fundamental knowledge about these kinds of questions so we ask questions about how people communicate with weather information and then underlying that is really trying to understand how people perceive and respond to weather risk because that is what frames how they take in new information and then to use that knowledge to try to improve weather risk communication and I want to emphasize that that sometimes when we talk about communication people think we're actually we actually communicate in practice so there's communication practice how to actually communicate and a lot of the work I do is really about research to build fundamental knowledge to try to understand these things so what we see now in the community is people doing a lot of kind of piecemeal studies to try to address one problem what should this graphic look like what should this warning system look like what should this product look like and so on and so really what my goal is to kind of look across problems and build fundamental knowledge so that we don't just fix one problem one place to see it pop up somewhere else which is really what happens often as you see there's been a lot of talk about communication and uncertainty whether service has been working on it for years it's really to try to take a broad view of this and build fundamental knowledge and so we use theories and methods from social sciences from different social science fields and integrate that with weather prediction knowledge and so that I think is one really strong point of the work that we're able to do here at NCAR is that we're able to integrate these two there's a lot of interest in this area as I mentioned and some people are doing work that's kind of weather prediction related but they don't know the theories and methods to use and other people know the theories and methods they're social scientists but they don't really know what's practical in the atmospheric science community so we really try to bring the two together okay so next I'll talk briefly about some research we've done on flash floods recently and so this was part what I'll talk about today was part of a much larger project that was funded by NSF where we had a multi-method investigation of the warning system and this was all done before the flood in 2013 so all of the what I'm going to talk about today are people's perceptions and their information used in someone before the flood there's two major parts of the project I'll talk about today one of these what we call mental models interviews where we do those with professionals so national weather service forecasters broadcast media and public officials, so emergency managers in the Boulder area and also members of the public and so the idea behind these mental models interviews is you use this kind of open-ended and increasingly structured interview protocol to elicit what people think about the risk unprompted by kind of your own the expert, the interviewer's ideas and so you learn what different kinds of professionals think about the risk and how they use information and you learn what different members of the public think and kind of put the two together to see where are their key gaps some perhaps some areas where members of the public don't understand something that would be important for making decisions when a weather event threatens and we also did a survey of Boulder residents and so we addressed several research questions in this study but the one I'm going to focus on today is how do professionals and members of the public conceptualize flash flood risk and make warning decisions on the right-hand side here you can see some pictures that I pulled from the 2013 flood and this is just to give you an idea of some of the life-threatening situations that people experience and the diversity in them and I'll come back to this later okay so one of the things that we learned from our studies is that some people have misconceptions about some key things about flash flooding not everyone but some people and so this is just one example here we also have data from our survey that suggests the same thing that we have in our models interviews so for example in these interviews we identified 250 some concepts that were mentioned by the professionals and then we identified which professionals mention those things and then which members of the public so these are some examples of some of the codes we have you can see this idea of happens quickly or lack of warning this is a key aspect of a flash flood that's why it's a flash flood not a flood and all the professional mention this so they know flash flood happens quickly if you look you can see that only about anywhere an hour, an hour and a half interview mentioned this idea about a flash flood happening quickly. So they just didn't get the idea about a flash flood. And then if you look at the speed or force of water flow that's another important aspect of a flash flood that the water moves quickly and so that's why one of the reasons it's so dangerous and so on you can see that some members of the public just don't understand this. So this is an example of one quote from a public interviewee when this person was asked what we mean by imminent. This is what they said. It could be a few days or a week. So you can imagine if this person gets a flash flood warning they're not gonna act now, they're gonna wait a few days or a week. They're gonna think they can wait to get more information. And I mean I should say that when we look across the data no one had all these misconceptions. So everyone had some piece of it right. They might know that the water would move fast or might have carried trees but they didn't know what happened fast or they kind of just had incomplete understandings that could really influence their decision making. So this is another example about one person who said talked about what you would do. They say you'd get supplies, you get a life vest, you get a boat and you'd have swimming skills and so maybe it would help but it's not really what the professionals recommend because you might get hit by a giant tree or rock. And so we talk about this in our papers about kind of how people's perceptions of the risk and their understandings interact with their decisions and what that means for how you communicate about the risks. So now I'm gonna talk about another aspect of the study which is that we have a lot of data on this from our mental miles interviews also but here I'll talk more about the survey. So we had a question on our public survey that was if you hear a flash flood warning you should blank. And people filled in the blank so they wrote their own statement. And the classic advice is that you climb to higher ground. So you can see that people said this kind of thing. Climb to safety, go to higher ground. About 85% of people said some version of that. And of course these are people could give long answers so some people said more than one thing. So then if you look at the data in more detail you can see that people have these different ways of talking about climbing to higher ground. So there's get to higher ground and hold on to what I don't know, you know, to a cliff, climb a tree, run like nuts. And of course the famous get as high as possible and not just one person said that. And remember this was like five or six years ago. So it's you know, anyway, but their idea I think is really that you need to go up and and anyway, so and then some people talked about moving to a safer area or staying out of dangerous areas, which is also a good idea. Some people talked about being careful and knowing what's going on around you and that was really a key element that came out in our mental models interviews that the professionals talk about the importance of situational awareness because the situation can evolve so rapidly and you might not get a warning. And then other people said things like this, it depends on where you are, like I don't really know what to do or have high ground pick out and go to it if you see something coming, think, assess the vulnerability of the location and act accordingly. And these are all right answers to and this kind of indicate the complexity of decision making. And so this really came out in our study about how because flash floods are so spatially variable and complex, and they evolve so rapidly that the professionals are often behind in the situation, they don't know what's going on, they're making their best guess about it. And this really leads to a lot of complexity uncertainty and protective decision making. So if you go back to those pictures I showed you about the flood, you can see that people in these different situations would have had to do very different things. So and on the one on the left, if you weren't driving on that road, then you were fine. But if you were driving on that road and it was dark and you couldn't see it then, that wasn't so great. In the middle one, you can see that, you know, part of the house is gone and there's a tree that's fallen over. So if you're in that tree, that wasn't so great either. And then on the right hand side, you can see that the road is in a bad shape, but the houses behind are totally fine. And so I was involved in the National Weather Service Assessment after the flooding we had in 2013, where we interviewed a lot of emergency managers and forecasters and so on. And one of the emergency managers talked to us about how they had people calling them during the flood and saying, what should I do? And they said, I don't know. I don't know if your road is still there. I don't know if you can drive out. I don't know if you have time. I don't know if your house is okay. I don't know if you should climb up hill or the hill is going to fall down on you. They just couldn't, they couldn't tell them what to do. And so yeah, yeah. Yeah, but you don't know, right? Yeah, I mean, I have a friend who said the same thing. She was driving and she heard the warning. So she drove to a high spot and I said, what do I do now? I can't see anything. I don't know what's happening, right? So and so this is kind of indicative of the complexity of the situation. I know there's a lot of research going on and current elsewhere about how to improve the forecast and the warnings, kind of from the meteorological and hydrological perspective. But really what we've started to think about is what does this mean for creation and communication of warning information, especially for rapid onset events. Now flash flooding is one of the most complicated weather type events, because the interaction with the land service makes adds a lot of complexity. But for tornadoes, there's some similar issues in terms of the complexity of decision making, depending on your situation and lack of really time to assess the risk and the combination of visual cues from the environment. Are you going to get those? Would they tell you all those kinds of things? So this is really, I think an important area for future research that we're starting to think about after the study and after this event. Okay, so now I'm going to switch to talking about hurricanes. And so in some of our studies, we've looked at different kinds of events like a flash flood, which happens more quickly and a hurricane, which has a longer time period to compare and contrast what happens when you have events with these different kind of predictive periods and characteristics. So I'm going to switch to talking about hurricanes. So we have several studies on this. And some of the kind of research questions we ask are about how people obtain an interpret information about an approaching hurricane and use it to make protective decisions, especially evacuation. And then more specifically, which types of messages help people take appropriate protective action or not and why? And so I'll only talk about a subset of our work here, but I'm going to focus more on the messages and how people use that information. But it's important to know that this is on the context of how people make decisions more generally. So when they get information about a forecast, it really comes in in combination with everything else in their lives, especially for a hurricane. You know, how much money do they have to evacuate? How do they perceive risk? Do they think hurricanes are risky? Where do they live? All those kinds of things. And so we try to think about it in that full context. And there has been a lot of research over the years, the past few decades in evacuation decision making for hurricanes, but only in the last five or more years have people really focused on forecast and warning information. So if you look back ten years or more, that was really people started to say, oh, it's not just about evacuation orders and people know if they're in evacuation zone. It's really people evaluating the risk for themselves. And so that's kind of what we're looking at. So I'll talk about results from two studies. First, I'll just briefly show some results from interviews with coastal Texas residents that were affected by Hurricane Ike. And this is work with Mary Hayden. She and I went down to Texas, the Galveston area, after Hurricane Ike and did some interviews where we talked to people who were in areas that were seriously affected. And I think this is really valuable because the next study I'll talk about, which is a survey, we asked people in a kind of idealized context how they would make decisions. And that's to kind of work in an experimental context where we can control variables and see what happens. But it's really important to think about this in the full complexity of decision-making because when you talk to people about their stories, about how they evacuated and why, there's a lot of complexity and there's a lot of kind of information there. So that kind of full complexity of the real world is what informs our more simplified experiments. Okay, so Hurricane Ike happened in 2008 and it was a large category two storm that large, like in size, and it created a 20-year more foot storm surge. Well, it was predicted for the Galveston area. The storm surge in Galveston ended up being not as high as expected. So the Galveston seawall protected Galveston from the storm surge. The areas kind of further to the east did get higher flooding and then in Galveston what happened is beyond the seawall and then on the other side of the island the water came in. So the water just came in from somewhere else. So it didn't go over the seawall but just came around the other side. And so before this storm hit, because of this 20-year more foot storm surge forecast, the local national weather service, they got really concerned that people weren't evacuating in the Galveston area. And so they issued the statement that that person's not heating evacuation orders and there's a lot of language in there about who and exactly what will face certain death. And so this was widely reported in the media. And actually after the first issue in they changed it from will face certain deaths and we must have thought that was an overkill. So they changed it to may face certain death. So people got this statement. Anyway, so the part I'm going to talk about today, we asked people about this statement. You can see that, you know, yeah so you can see this kind of in the context of the 20 no statements I talked about earlier that kind of what's going on now in terms of communication. Okay so this is what happened in Galveston. These are some pictures that Mary and I took. The one on the left, you can actually see the seawall right here on the edge. This is the Galveston seawall. So this is on the on the water side of the seawall. You can see that this building is totally devastated. And then this is kind of in the middle of Galveston. You can see that there was flooding everywhere basically. If you walk around Galveston Island, some houses are an eight or ten foot stilts. They were moved up after, you know, the hurricane from a hundred or so years ago. And the ones that aren't some of them were totally flooded. And people just a lot of people just hadn't expected this. Okay so I'm really going to focus on though our questions on the interviews about the statement. And so we asked people if they'd heard the statement prior to Ike. And if they had heard it, did it affect their decision to prepare or evacuate? And all of these people were in high risk areas. So Galveston and then the areas that were harder hit in the east. So about a third of the people said they didn't hear the statement. And from those that did hear the statement, everyone remembered it. It wasn't like they weren't sure. They really had a strong opinion about it. We didn't really have to ask the next question about how it affected them and what they thought about it. So of the people that heard the statement about a third of those said it affected their decision. And the other one people said it didn't affect their decision. In most cases because they'd already evacuated when they heard it. So it didn't, because they'd already decided what to do. In some cases it was because they'd already decided to stay. Nothing was going to change their mind. So for people that affected their decision, all of them said it helped them decide to evacuate. So in some cases it was say a husband and wife. And one person wanted to evacuate. The other one didn't. The statement came out. They said, oh, listen, this is really bad. We have to go. So everyone that said it affected their decision said it did help them evacuate. So in that sense the statement was a success. But then as I talked about earlier, you know, in Galveston there was a lot of storm surge. But people didn't, people stayed, plenty of people stayed and didn't die. And everyone that we talked to afterwards knew this. Some of them had stayed and didn't die, although some of them were in situations that were less than ideal, but they were sort of life-threatening, but, you know, but they didn't die. And so we wanted to know what people thought about the statement in retrospect. So we asked about what their opinion of the statement was. And as I said, people had really strong opinions about it. So about half of the people had really positive opinions that scared you to death. It made you realize that if you didn't evacuate, you're an idiot, you know, those kinds of things. And the other half people had really negative statements. Like it was overblown, it was rude, you know, one person even said it may want to stay and show them that I was going to be okay. And so kind of this, you know, I mean, that person was going to stay anyway, but still you see this kind of response. And so I'll talk about this in a minute, that there's theories and risk communication that talk about these kinds of messages. They're called fear appeals, that if you are too sort of fearful, then people will, some people will really reject the message and engage in emotional management behaviors rather than actually taking the protective action. And this has been shown in plenty of health context. Sometimes actually people engaging in the action more. And so, you know, this is some evidence of that kind of thing might be going on. And so when people talked about the statement, you know, really, as I said, it didn't convince anyone not to evacuate, but we don't really know how it's going to influence their opinions next time it happens. And then overall, the story from this, as I mentioned earlier, was that many people didn't adequately prepare for flooding. So this is an example of one person who said I never dreamed of seven feet of water. People we talked to, some people had moved things higher in their home, they moved things to higher levels on their bookshelf. Everything was flooded in their house. And it was really tragic to talk to people who really, they prepared their house for wind, but they just did not expect flooding. And this is known that people don't really understand the risk of storm surge. People had a lot of reasons for not understanding it. But still, it really points to the importance of communicating these kinds of risks better so that people can prepare for the right kinds of things. Okay, so next I'm going to talk about a survey that we did of coastal Miami residents. And this survey was done in 2011. So in this survey, we tested a bunch of different things. But the part I'm going to talk about today is testing different messages, people's responses to messages. So for this survey, we had about 260 respondents in areas at risk of storm surge. So in this case, we focused the survey in one area in Miami, because we wanted to be able to present them with a scenario and have people be at similar risk. So we wanted to kind of control for what level of risk they were at sort of geographically along the coastline and also as you go inland. So we targeted people in evacuation zones with the idea that in this scenario, all of them would be at high risk and all of them should evacuate according to emergency managers. And so you can see the evacuation zones here. This is our region of Miami that we targeted for our sample. And then the survey was implemented online with mail recruitment in English and Spanish because there's a large Spanish speaking population in Miami. And so this study was also funded by the National Science Foundation and NOAA. And so the survey was developed using a combination of different things. Some of it was following up on the ice that I just talked about and also on the password that Jeff Lazo and other people have done. And then we also brought in some risk communication theories. So our goal was to use some theoretical approaches from risk communication to ask questions to understand why these kinds of things are happening, like I just talked about with the responses to the messages or why some people evacuated in some don't and things like that, how people are getting information and so on. And then we also had an expert advisory group that consisted of researchers and national weather service forecasters and people at the Hurricane Center, public officials and members of the broadcast media that gave us ideas of what we should test based on what they were seeing as far as communication issues in practice. So we brought all this together and I only talked about one part of the survey which is a section where we asked people's responses to test messages. So in this part of the survey, each respondent received a randomly assigned combination of messages about the same hurricane scenario. So the National Hurricane Center developed a scenario, they actually had a scenario that they were using already for an emergency management exercise in Florida. So they developed products leading, you know, going forward in time for that scenario, including the cone graphic and other things. And then the weather service office in Miami also developed products. The products that they would typically develop for this kind of scenario for us that we could use these to develop our messages. And in the results I'll talk about today, this is what people said when we said the storm is going to make landfall in about 48 hours. Okay, so people received different messages and then all respondents were asked the same set of questions. They were asked about their intent to take protective action. So their intent to evacuate. And we had several different other preparatory actions like buy supplies, you know, shutter your house, things like that. They were asked a lot of questions about their perceptions of the risk, their perceptions of the message, their perceptions of the source, and so on. And those were based on these theories that we were trying to apply here to try to understand the kind of dynamics of the decision making. And then we also had questions about socio-demographics like age, gender, and so on because we know that those are influenced people's protective decision making. World views from the cultural theory of risk, this is work that we're doing with Heather Lazarus where she's brought this into different contexts where these world views have been found to be important in terms of how different people make decisions and view risks in different contexts we brought that in. People's hurricane experience and the perceived evacuation barriers and resident safety because those are all things that have been found important in past research. And then we also had yeah? No. Yeah. So I'll show you that in just a second. In fact, this is the next slide. So we had five different elements of the message and people received randomly assigned combinations of those. And so all respondents received one graphic. They received this cone of uncertainty graphic, which is kind of the standard thing. And in Miami people are familiar with the kind of graphic although we're not really sure how they interpret it but they're familiar with it. And so everyone received this graphic either with or without that track line that you see. And then we had these five other textual messages and people could either receive or not receive each of these other messages. So they could receive any combination of these. And you can see different kinds of messages. We were interested in communication of uncertainty. So that's why we had this graphic. And then we had this 55% chance of the storm will make landfall. We were interested in storm surge communication. So we had a message that was about storm surge kind of the science or factual piece of it. And then we had a kind of more dramatic message about storm surge kind of similar to the certain death message or kind of the kinds of things that people are using. And then we also had what's called an efficacy message that talks about evacuation because the research that I mentioned earlier on fear appeals talks about the interaction. If people don't feel like they can do something, then they're more likely to have these fear reject the message responses. So we included that as well to see the interactions between the fear message and that. OK. So the data analysis I'll show today is multiple linear regressions where we ask how the likelihood of taking protective action varies with all the variables I just talked about. So we did the sociodemographics and the worldviews and so on. And some of those are significant, but I won't talk about those today. But we have a paper if you're interested in seeing it. And then the experience and so on, those other variables we included all of those. And so what I'll talk about now is just controlling for all of those things. How do people respond to the messages that they received? And then I'll talk briefly about some other responses to these test messages because we're interested not only in how people behave, but like I said, during the Ike scenario, people also have other kinds of responses to the messages that might lead them to trust someone more or less next time. OK. So this shows the results from the portion number of regression of with the messages. And so if there's a plus on the right hand side, it means that the test message positively influenced that dependent variable. And if there's a negative, it's the reverse. So you can see on the top. So on the left of the middle column here, this is evacuation likelihood. So these are the main results that we really talk about is what made people, which messages made people more or less likely to evacuate or to intend to evacuate because this was a hypothetical scenario. And then we also have these other protective actions. So I've just indicated here the signs kind of across those. And so our biggest signal was from this storm surge, dramatic storm surge message where it really made people more likely to evacuate. We thought was where we saw our strongest signal. We do have some other signals in this data. For example, the cone with the line, people who received that were more likely to evacuate than the people who received the cone without the line. That agrees with some other research by Bob Meyer et al. We think this is because people indicated that they didn't understand the cone without the line as well. This 55% chance that the eye of the hurricane will make landfall, that actually decreased people's likely to take protective action. So this fits with some other results by Sam Meyer, but from a different study where they saw that people really overestimated the risk of hurricane force winds leading up to hurricanes. So probably some of these people actually thought there was a greater chance of an eye-making landfall in Miami-Dade County based on the cone. Yeah, Linda? Yeah. Yeah, no, I think that's true. I think we... Yeah, well, I think we were kind of looking at something in the middle, so not too high. We actually thought this would increase people's likelihood to evacuate. So this was really... No one had really done this kind of message testing for hurricanes before, so we really didn't know what would happen, to be honest. So we were just trying some different things to see and we discussed this with our expert advisory group, kind of what the level of risk is and so on. And so people just talk a lot about communicating probability, so we just wanted to see how people respond in this information. And then we had this message about storm surge being four feet or higher and that increased protective actions. Of course, if we had different levels of storm surge, it would probably be different. And then, as I mentioned, this message really was our strongest signal. The people who received this message had a higher intention to evacuate. So that agrees with the ICRE results and it's kind of what people are aiming at with these kinds of messages. So just looking at some of our other responses to messages and this is just comparing people who received the message to people that didn't. So in terms of likelihood evacuation, as I mentioned, people who received the message were much more likely to evacuate. And we also got signals on some of these other measures that we had. This information is overblown. This is a measure from one of the risk communication theories but it's also overblown as one of the words that people used in our IC study when we had open in the questions. So people were more likely to think that information is overblown and they were less likely to think the source is reliable. So as I mentioned, this was a first test of some of these ideas and there's some things that are going on that are not exactly what we'd expect from the theory. For example, a lot of people thought the information was overblown but they still evacuated which isn't what the kind of risk communication theories might suggest. And so we can't fully kind of tease out what's going on with this sample here but we have some ideas and so right now we're working on some follow-up work with some collaborators at Rutgers who are putting together a survey with some post-sandy money that they have. And so we're working with them to test some of these ideas again on a survey and a different population in the Northeastern U.S. based on what we've learned kind of to target it more precisely. Okay, so in what I've talked about so far, especially the last part of the hurricane results, I really focused on how people respond to one piece of information. And if you look at kind of hazards and disasters research, people talk about this cycle preparedness and warning, response, recovering, mitigation and often they talk about the response in terms of response to an event. So after the event happens, not really looking at the forecast and warning phase. And if they do talk about the forecast and warning phase, they talk about one forecaster warning and how people respond to that. And so in today's world, especially for an event like a hurricane, that's really not what's happening. Now forecasts are good enough that five, seven, even more days before a landfall, there's information going out about the storm. So the storm is approaching and there's different information going out about the storm. There's different information about the forecasted areas of risk as the storm approaches. There's people of different types on the coastline that are getting this information, exchanging the information with each other and making decisions and deciding what they think about it. And so, and there are these complicated social information networks, for example, especially today with social media and the internet, people always talk to each other about what's going on. But now they can communicate information much more quickly in much more complicated ways. And so that's really changing how people use information of all sorts and weather information. And we really know very little about this. So in the next project I'll talk about, we're looking, trying to look at this whole system, the dynamic weather information system and looking at how the creation, communication, interpretation of use of evolving information about approaching risk how this whole system works. OK, so this is our project. We call it CHIME, Communicating Hazard Information in the Modern Environment. In fact, I should have told Morse to bring his bell so you could have had a little bell sound. So our goal in this project, which is a big mouthful, but as you'll see, we have $3 million. So we have to have a big mouthful goal is to investigate how interactions among actors and information influence how people interpret the risk and how they respond as a hurricane approaches and arrives. And the key of this study, I think what's really new that we're trying to do is to do this in the context of evolving meteorological predictions. So we're looking at it temporarily evolving as the predictions change and the information about the risk changes as the storm approaches and what we call the modern information environment, which is how people are exchanging information in the modern world or how a lot of people are, I should say. And so this is the grant from the NSF Hazard Sees Program. We're about a year and a half into our project. My co-PIs are listed on the bottom, including several here at NCAR. And for this proposal, you had to integrate concepts and methods from an expertise from three different NSF directorates. So you can see here the three different NSF directorates that we incorporated. And even if you look within those, within our collaborators, almost everyone's from a different discipline. And so we really have a lot, even within social behavioral sciences and computer and information science have a lot of different disciplines coming in and a lot of different perspectives. So as you can see with our methods, we're trying to bring it all together using different methods and the different kind of knowledge from these different areas to address this whole big problem. OK, so here's the methods. And I'll go over a few of these in a little more detail in a minute. But the two major things we're trying to do are modeling of the information system and empirical analysis or observations of the real information system. And we're going back and forth from the two. So with the modeling, we're able to look at things in a more idealized context. And with the empirical analysis, we're able to see the system in its full complexity or try to understand its full complexity. But it's actually, of course, really hard to understand its full complexity. And so the modeling kind of helps us understand what's going on in the real world. So we have different methods. So I'll talk about a few of them. In a minute, we have some Hurricane and Storm surge modeling. We have agent-based modeling that I'll explain in a minute. We have a part that's analysis of social media data. In this case, we're focusing primarily on Twitter that was collected during hurricane threats and other threats. Actually, we just collected a data set. We're collecting data that is at now from the snowstorms on the East Coast to see what's happening there. And our colleagues have done some work on the Colorado flooding. And then we have some other methods, these follow-up interviews with social media users. We also have some focus groups that Olga Wilhelmy and Heather Lazarus here at NCAR are leading, where, of course, not everyone's on social media. So we're trying to compliment that by looking at populations that are more digitally vulnerable or that might not be involved in social media to see how their risk interpretations and information they're getting might change. I'm looking at issues about vulnerability and adaptive capacity with those populations. Towards the end of the project, we have these prototype visual integrations of information where we're going to try to take what we've learned and use it to develop, to think about how to communicate the information. And then we have a bunch of stakeholder interactions to kind of keep our work relevant in the real world. So the hurricane storm surge modeling, this is work that's being done primarily at this point by Chris Davis, Chris Snyder and Kate Facel. And so the goal of this is to investigate the predictability of storm surge across a range of lead times. And in developing this project, one of my goals was really to have fundamental research that advanced the science and the knowledge in each of the three directorate areas and also to bring it all together. So this was the part that's kind of advancing the knowledge in the atmospheric science. So we're using the advanced hurricane work model coupled with the ADSURC model to simulate storm surge. And so right now, they're doing some work on predictability of storm surge at different lead times. And then later, we're going to couple with this agent-based model I'll talk about in a minute to study how the hurricane and storm surge deformation propagates to the information system. And the goal there is we can run some simplified experiments for things like say the forecaster this kind of accuracy or say we produce this kind of probabilistic information. What does that mean for the system and for people's protective decisions? Another part of our project is agent-based modeling. So what this agent-based modeling does, you can see this diagram on the right. This is a simulation for Hurricane Wilma in Florida. And these little dots are different agents, so different kind of pretend people. Most of them pretend members of the public. And the idea is to develop a simple model of social actors, so these members of the public. And they pursue process and transmit information. So they get information from broadcasters and media and so on. And they exchange it with each other. And they make protective decisions. They don't actually do anything. We're not doing any evacuation routing. That's what agent-based modeling has mostly been used for in the hurricane context. But the idea is that by doing this, by building this model, we can explore in this virtual simplified laboratory how the information, the information networks and the protective decisions are interacting. For example, we can create a world where there's a lot of density of social media networks and a lot of penetration of social media versus one that's not. And see how everything changes. And then also integrate with the spatially explicit hurricane and storm surge modeling to run coupled experiments, as I mentioned before. So we're doing the work separately. And then we're starting to think about how to bring it together. So the last part of this project I'll talk about is this analysis of social media data. So our colleagues at the University of Colorado, including Laysha Palin and Ken Anderson, work in this area called crisis informatics. They talk about this as how the study of how technology is changing how the world responds to mass emergency events. So they've done a lot of work in disasters before, hurricanes, kind of social events like protests. They've done some work on flooding. But really, all of the work that they had done when we first started talking to them was during and after the event, not in the forecast and morning phase. And so what we're doing is kind of taking the infrastructure and the analytical tools that they've developed and applying it to the forecast and morning phase as well to try to understand people interact with interpreting news information as a hazard approaches and arrives. So in this case, we're using Twitter data, they also combine it with other internet data. And of course, this is limited in the sense that we're only getting data from people who are on Twitter, but it's a freely available data set of what people are thinking in a temporally evolving way as the storm arrives. And so it's really potentially rich in that sense. So some of the challenges are that a single event can generate hundreds of millions of tweets. And basically each time they have a new event, they get 10 times as much data and they have to figure out how to deal with it. So they have infrastructure to collect the data, but they analysis is really the tricky part because there's a lot of data and it's awarded data. It's not numerical. And so there's a lot of complexity. So what people usually do is they say, oh, I'm gonna pick 500 tweets and analyze those or 500 users. And so you don't really know if you're getting a representative set. There's a lot of uninformative content in this data. People just talking about, I'm going to the bar because Hurricane Sandy's coming. And so that's informative, but you can't really get the richness of their decision-making. And so to answer behavioral questions, there's really a lot about that. We have a special set of the analysis is just that. But anyway, so to answer behavioral questions so in detail to understand what people are doing, you need detailed analysis on a small set of tweets. So part of it is kind of separating the wheat from the chaff and figuring out who is talking about what and yeah, how do you kind of look at this data? So just to give you an idea of the size of the data sets. These are some of the data sets we're working with. So we happened to meet with these colleagues for the first time right before Hurricane Sandy was making landfall, it was maybe five or six days before landfall. Julie, Jamith, and I were there. And we said, hey, there's a storm coming. Why don't you collect some data? Never done this before. And so if it doesn't make landfall, then just throw it out. So it turned out that it did make landfall. And so we had this Hurricane Sandy Twitter data set. And so you can see the original data set that they collected is 22 million tweets. So these are some of the ways we parse the data to try to kind of, that first data set contains Sandy keywords about like Sandy or other weeks we were talking about Sandy. But people, as I'll show you in a minute, people talk about Sandy without saying Sandy. So they went back and collected for all of these users that were geo-tagged on the Eastern seaboard of the US. They collected all of their tweets during the periods. You could see their full kind of decision-making, even if they didn't explicitly reference Sandy. And then we narrowed that down to a smaller time frame and a small geographical frame to make it more manageable, but it's still 700,000 tweets. So you can't look through it by hand. I tried. I can tell you that. So what we're doing is this interdisciplinary multi-method analysis of the tweet content along with the metadata. So they have several grad students who are focusing on different things and looking at things like, for the geo-tagged tweets, can you see people's movement and kind of understand how people are moving, who's evacuating, who's not, and things like that. And so one thing they've found so far is that people evacuate, they often don't tell you. So you can learn about it from their movement, but not really from what they say, or they say they evacuate, but they weren't geo-tagged. So we're really trying to bring this all together. The project also includes linguists. So the idea is you look at this kind of, you kind of go in-depth in certain, in kind of individual Twitter streams of people and what they were saying is a strong approach. And then you use that to develop automated algorithms to tag things, different kinds of things that we've found are interesting in the data. So both so that you can take an aggregated look at what people are saying and also so that you can identify who are really interesting people to look into in more detail, people that may be changed with how they were thinking about the storm as it evolved or so on, and how they're using information and all those kinds of things. So this is just one example of Twitter stream I pulled out. You can see this person had only one geo-tagged tweet, that's the one with the star and that's how this person ended up in our dataset the rest of the time she wasn't geo-tagged. You can see the reason I pulled this one is this tweet in red. She talks about being in an evacuation zone and what did she do. She never actually tells us what she does, but you can see from the tweets that she ends up staying. As you can see, there's some tweets in here that don't have any relevant content, but there's also tweets, like she watches them up at movie, I mean, that's interesting, but I think she watches them up at movie during the storm. So yeah, there's a lot of these kinds of things in the data like you just don't even know what they're talking about. So yeah, so you can also see that there are tweets like this where she doesn't mention Sandy and so we wouldn't have collected that if you were looking for Hurricane or Sandy or anything else. The only reason it's in this dataset is because our colleagues did this contextual data collection where they went back for these users and gathered everything they were saying during this time. So you can see as kind of going through the storm first she talks about it then she says, what should I do? Then she talks about and she even gets us pictures of storing her supplies away from the kitchen window. So she does take some preparation action. She looks outside and takes a picture of these branches, tangled them power and cable lines and she talks about being nervous so that that's indicating that she's getting a little bit scared maybe and then the roar of Sandy is terrifying and setting up at the same time. Then she eats, then she watches the movie and then in the end she tells us that she's alive and has power. So this is the kind of thing we're trying to look at and try to, we have 100,000 of these people and so the idea is to kind of figure out what they're thinking about and some of them talk about using specific information or make reference to hurricane information or evacuation orders. So from that we can learn how they're using different kinds of information. Okay, so like I mentioned, we're in this project, the Hazard Seeds Project, the Time Project, we're looking at this whole system and we're doing it through a combination of modeling so we can look at the system in an idealized sense and kind of full observations, real world observations of the full system and we're also interacting with stakeholders to kind of make sure that our more idealized research is relevant to the real world. And I think this is an example of kind of the direction I wanna go in where now that I understand things on the kind of communication and use of information side, you can really bring it all together and integrate more fully with the atmospheric science work. Okay, so going back to the beginning, I talked about kind of my career path and how I've done different kinds of things and bring it all together. And so I think that as I mentioned, my goal, a lot of what I'm doing now is really trying to bring all these different things together and I talked about hazardous weather information in my talk today about I also do some work that kind of bridges into the climate time scales. For example, we have a new ESM-3 project where we're looking at decadal prediction, we're trying to bring these kinds of things together again and so that this kind of way most of what NCAR does, rightfully so, really focuses in on this kind of the science and the observations and the prediction and the modeling and all those kinds of things. And so I think what I work on really is kind of also bringing in the expertise on the other side, the other side of what I'm talking about here and how that information is then used by people along with my knowledge about the atmospheric science to kind of make that information on the atmospheric science side more relevant and useful and also to look at the whole system. So look at the atmospheric science information really in its larger context. So moving forward, as I mentioned, a really key area I think is integrating physical and social sciences to learn how to improve creation and communication of different kinds of information. Particularly in the community now there's an emphasis on communication of uncertainty and also perhaps of certainty and that's an area I've worked in in a long time but it's kind of coming back into fruition after maybe less interested in it for a while. People talk a lot now about communication of impacts in the weather community and then sometimes we'll talk in terms of risk rather than impacts and so there's a lot of talk about it but really not much knowledge about how to actually do it yet. So that's, I think, an important area for future work and to do all this using understanding of how to perceive risks, how they interpret information, what they do with the information, how they make decisions and also as science and technology advance. So for example, that ESM-3 project I mentioned is really about decadal predictions and so as science advances and gives us new capabilities like better storm search predictions and so on, kind of what do we do with that information and how do we ensure that it's not just lost in the sea of stuff that's out there in these days. And as I mentioned in our time project, really working with people who understand information technology to see how this information kind of gets used in this modern world that we don't really understand very well and as our colleagues in information science talk about, they don't even really try, the system moves so fast, you can't even really understand it. So you try to understand it and you realize that people out there are engineering the system in a way you haven't even thought about. So you're kind of just always moving forward and kind of recognizing that's how the world is these days. So in this diagram, you can see kind of examples like ensemble hurricane predictions and you can use those to predict impacts or risks that people care about. So not just predicting the weather, this is a real emphasis of the community, it's predicting impacts and then giving that information to different people who are all interacting and might interpret that information differently. And really the key is not just doing that but bringing our knowledge in about what people are gonna do with the information to figure out what kind of scientific predictions to produce and what kind of impacts to wanna predict and really how to do that. So bringing it all together. And then as I mentioned, another thread running through my work, even going back to my undergraduate work and my undergraduate thesis is really combining investigations in simplified context. So it could be modeling or it could be the kind of surveys or experiments we do where we test kind of something simple. You give this message, what do people do? And to combine that with analysis of real world observations so you can understand how it fits in the real world and you can use the real world observations to kind of motivate your simplified investigations and also understand them much the same way as we do in kind of weather prediction research. And then using a mix of theories, concepts and methods. So for me, I'm not trained formally as a social scientist but I work with a lot of them. And for me, when I come up with a question about a problem, it's kind of like I can go out there and see who's the best person to work with on this, what kind of methods and tools we wanna bring in to really understand this. And so work with a lot of people to bring in what's most appropriate for the kind of question that we're asking as opposed to just saying, oh, let's do this because it's what we've done before. So I will close there and hopefully have time for a couple of questions. For questions, I ask you to get the microphone so it will be recorded properly. I was intrigued by this death is certain message that you mentioned a couple of times. What's the thought process now on that kind of message? One of the reasons I asked is on the north side of the Boulder Reservoir, there's a trail that has a sign that don't wander off the trail, death is certain. And I laughed the first time in fact, I wandered off the trail to get a good picture of the sign. So, isn't that a bad thing to say or could you elaborate a little bit more on that topic? I think it depends. I mean, it can motivate people, especially if something is staring them in the face. And you think in that case, there's not really any sort of imminent risk. So in the hurricane case, we've seen that in the face of that people, it does give people's attention and they might pay attention to something that hadn't before. But like I said, there's this whole research on health risk communication and actually in climate change also that these kinds of fearful messages off and they turn people off and they lead people to engage in these other kinds of responses. Like you did, like they do the wrong thing or in your case it was okay, but I mean, they do the thing you don't want them to do. Like that classic example in health risk communication is this, well, some of you probably remember this, you know, your brain on drugs with the egg in the frying pans like that. So it seems on the face of it that if you scare people, they're gonna do what you want. But it's not true. And so really what we're trying to understand is kind of how that works out. Who are the people that go in different directions given this messaging? There's another anecdote about that from the Arrino tornado. Yeah, yeah. Where some broadcasters in Oklahoma City said, death is certain, you better leave if this tornado is coming into your neighborhood. And so a whole bunch of people in their cars left and then died in the flash flooding that happened a couple of hours later. So unintended consequences. Like sometimes the messages aren't geographically targeted but sometimes they are. So if you read the full certain death message, it says if you live in a single story home, if you live here, but nobody reads all that stuff. So kind of some of it's about the message targeting and you know, in hurricanes you get this overspray of messaging and people evacuate that shouldn't and then you get traffic and all those kinds of things. So in comparing flash flooding with hurricanes, there's obviously different time scales and predictability and all that. And for the hurricane studies you were looking at, people's, what people were posting on Twitter, what they were saying about it, what actions they were taking. Have you or any of your collaborators looked at Twitter information from flash flood events? Because specifically for the Boulder event, information from Twitter was critical in just notifying everyone on what was actually going on. Whereas for the hurricanes, it seems more of a reactionary type of database. But have you looked at it from Twitter as providing an actual source of information that people may or may not take action on? Yeah, so I will say that there actually is a lot of information exchange as far as people providing information in our Hurricane Sandy Twitter data set. In fact, that's what our colleagues are really focused on in their post-disaster work before. Sort of what makes information reliable and who, you know, all those kinds of things. So there is that, but in a different way, like you said. So our colleagues at University of Colorado did collect a data set during the, and during the flash flooding in Colorado. They have done a little bit of work with it and in some ways it's easier because it's a much smaller data set because it's a more geographically focused area. They haven't looked at these kinds of things yet that we're talking with them about because for them they don't really think about protective decision making in this way and so they're learning from us about how to think about it, but the idea is as part of this project to use that data set to look at those kinds of things. They've really focused on, the other thing is that they didn't start collecting too early in the event, in part because it wasn't predicted that well. We didn't identify it beforehand in a part because they were actually in an area where they were worried about flooding. So I don't think they started that early, but there were still warnings going on through the phase that they collected data during. So yeah. And especially when you talked about the surveys you conducted, when you went to people's houses and talked to them after the impact about what influenced their decision most. Do you somehow categorize your data according to how much you trust this story? So something because you're not only dealing with a super complex system of a human being, but you're dealing with a human being that is in a hazardous situation and maybe more or less out of its mind or something. Yeah, so we actually have a lot of data on that both from our interviews and also from our surveys about how people trust different sources. There's a lot of interesting stuff. I mean, when you ask people, some people and other research has shown this too. If someone says to evacuate, there's an evacuation order from the public official, they just go. But a lot of people don't. Some people trust the National Hurricane Center, some people trust their local forecaster, some people trust their family. Some people have reliance on different sources. In our survey, the one I mentioned where we tested the messages, we also asked questions about, maybe it was a different one, but we asked questions about how much people rely on different sources of information for different kinds of things. And interestingly, I can't remember, it was 30% or 50% of people said they rely on the Hurricane Center and the Weather Service for evacuation information. So like the Hurricane Center and the Weather Service, in the case of hurricanes, has this kind of stature. And so people are relying on them for information that they're not usually the original ones providing them. So there's a lot of stuff about trust and sources. We have some other analysis that didn't show today where we do show that trust and source is an important predictor of what people decide to do if they decide to use that information. And I think it's a really important area for future research in the weather context. There's a lot of research in other contexts about the importance of trust and what creates trust and all those kinds of things. But we haven't looked at it that much in weather. Okay, we have a question from Rich, Roy, and then Ritt. And maybe Linda. What probability is that? 55. Well, you got a particularly long question though. It's going down. No, the whole science of trying to extract data from millions of tweets and this sort of thing, crowd sourcing, there must be a lot of kind of knowledge someplace like Google or the NSA or something like that. I mean, how much of this is accessible? Or is it all proprietary or top secret? Yeah, well, I think that people aren't using it to look at these kinds of questions. They're certainly doing, there's a lot of the people are doing for marketing kinds of purposes. But are they general techniques? Yeah, and so our colleagues at University of Colorado, they actually have a new relationship with Twitter so that they can get more metadata and they can actually go back and collect data for certain. So the problem now is you have to collect data in real time and there are some kinds of metadata that you can't get with the data that Twitter keeps in house. So they have a new relationship with Twitter, they can get that kind of data. But I think one thing is that people aren't really looking at these kinds of questions yet, but actually my brother just started working for Google recently and I've talked to him about, I know they have better analytical tools. There've actually been a couple of studies where people have used Google type data to see what people are searching for the hurricane approaches. And they say, okay, when the threat moves, people in that area start focusing on it, but I'm sure they have better analytical tools in house. They've just never focused on these kinds of questions because they're not really interesting unless they're trying to see who's gonna buy soup and market soup versus boards for your house or whatever. And so we're definitely looking into that and I think that's an interesting area kind of going forward. But we also wanna remember that not everyone is involved in this kind of Google, Twitter, whatever world. And so that's really what we're trying to do with those focus groups in that project is remember that there are these people who might be left behind, maybe they aren't, but kind of make sure we think about what happens to those kinds of people. Yeah, Rebecca, I was interested in your Miami hurricane survey and you asked some specific questions and you just said that the answer that you had a destructive storm surge was the most likely. It seems to me though that you have a lot of information, 261 respondents, could you come up with a better response, a text or a graphic? For instance, you said likelihood of coastal flooding. Well, Florida is all coastal. The whole Florida is likely to flood. Is there a way to come up with a map, for instance, that says where are we most likely to have coastal flooding and does that come out of a survey like that? Yeah, so in this case, we were focused on people that were right along the coast in evacuation zones which is why we didn't focus so much on where in the coast. So some of these things were following up in the survey that I mentioned that we're collaborating with other people on and there actually is some fair amount of work going on in storm surge risk communication in terms of using different kinds of maps and inundation maps that Jeff Lads and others have been involved in. I think that what we still lack is understanding of kind of what people don't understand about storm surge and we also had a section of our survey where we tested some of the messages that were being produced by the hurricane center and broadcasters and so on and we found a lot of people misunderstood them and we have information there about why that could be used to develop new messages but I think from that survey, we didn't ask, for example, about our messages open-ended questions so we can't really understand what we're really thinking about those kinds of things. So I think there is work going on in that area but then a lot of it focuses on what color should this graphic be or what language should we use and not really what are people misunderstanding and what is the most important thing to communicate. So we're kind of trying to do that side and then as I mentioned in our Hazard Seeds Project we have this piece that's called Prototype Integrations of Information but really that's more about that kind of communication where we'll bring in test different maps or use the different GIS layers to show people say inundation compared to different kinds of things and we test them in focus groups to get responses where we can hear what people are thinking, what do they understand, what do they misunderstand and those kinds of things but that's still a couple years from now but it is a big area of research, I mean big for this field which is not that many people but a big area of research, ongoing where people are trying to test different kinds of things. Rebecca. Yes. So very nice stuff. Thank you. I'm interested in, of course, I'm kind of more on the longer term climate side but as we discussed long ago there are all sorts of connections here. One of them is the effect of mistaken or deliberate suppression of uncertainty to try to use that as a motivator and so for example, let's take the East Coast storm and I'm sure you looked at more a literature or public announcements and so forth than I did but I did look at that and it was very striking to me that no public official expressed anything about the uncertainty about this storm and then you have this situation where the mayor of New York is criticized for having stopped the subways so it's the problem both of, was that a deliberate suppression of the uncertainty and then also the problem of what do you do about the false positives that are bound to come up and it seems to me that that suppressing the uncertainty essentially guarantees or makes it much more likely that people will perceive it as a false positive and won't take shelter next time. Yeah, I think that's true. I mean, we've seen that for example in Hurricane Irene in New York that some people didn't want to take action before Sandy because Hurricane Irene wasn't as bad as expected and I think there's a lot of different things going on. I think that at least in the weather community, I don't think it's, I think sometimes the message about the uncertainty was there somewhere, but it just didn't make the headlines and everyone sort of forgets about it. So that's a big thing. I think also the uncertainty formation is hard to communicate in a way that people can understand because it's so complicated but also that meteorologists usually talk like meteorologists. They don't talk like anyone who's not a meteorologist can understand and so that's a challenge. In our, in the mental model study that I talked about with flash floods and we also actually have a parallel study in hurricanes I didn't talk about today but we talk in there about how the forecasters and the broadcasters and the media, kind of how they think about uncertainty and how they use that in their decision-making and how their trade-offs between false alarms are protecting the public for them given their given goals. So we kind of look at the dynamics of that system. I don't think we know what the answer is yet but it's definitely, I think we have some understanding of kind of what's going on at least in the weather system and I think at least in the weather timescale people are all, their goal is really to protect the public and so they do it all with that in mind and they don't, in the moment, they don't really think about the consequences over the long term of thinking about different things because they don't want to be the person who did an issue of warning and someone died or whatever and so I think those are really important areas to look at and I hope that we can do that. I feel like there's maybe a lot of emphasis on just false alarms as false alarms and not really thinking about kind of the range of uncertainty that goes along with that so yeah, I think there's so much to do with that and you know, on weather timescales, climate timescales and kind of bridging between them. So we have some knowledge but really don't know the answer yet I would say. Your references to certain death got me thinking about crime. Yeah. And it seems apparent that some of this information intended to protect people can be used for nefarious purposes. Does any of this show up in the social media and has it been studied? We haven't seen it, there is a lot of conversation about people being afraid of looting and in some circumstances there are looting and people do stay and protect their homes because they're afraid of looting and that shows up in our data as one of our kind of evacuation barriers who want to protect their homes. I haven't seen it in the Twitter data although we might run into it I guess. You mentioned crime and this, the project I mentioned where we did the mental modeling and the survey on flash flooding was funded by the NSF Human and Social Dynamics Program and I went to one of the PI meetings and there was someone talking about crime there and kind of predicting where crime was gonna take place based on, it's actually you can predict crime based on like where sort of people of a certain age live and they always do crime like near their aunt. Some would, they always have a home location to bring their stuff to. So people are thinking about that kind of stuff but not so much in the weather context. So I think it's an interesting idea. I think that, yeah I don't know what the answer is and we don't really want people to stay because they're afraid of crime and so I think that that's, you wanna just make sure that the public officials can protect the locations and assure people that they will but it's something interesting to look at. Well I'll see if it turns up in our Twitter data. People probably aren't tweeting when they see crime but maybe they are. Are they experiencing that? Okay. Yeah, they might be. Let's thank Rebecca once again. Thank you.