 We're going to talk about decisions. But before we do that, how many of you all know one other person in this room? OK, so most of you have met one other person, three other people, before you came here. So a couple of people, more than five? OK. And then that's helpful to just kind of get a sense. So a lot of you all are meeting each other. How many of you all have ever taken a policy analysis class or a decision analysis class? OK. And then how many of you all would classify your research as being focused more on understanding and forecasting to better understand how the world works? More excited to the basic science piece of it. How many of you are more focused on forecasting to directly support decisions and you know what those decisions are? I think, Allie, is that right? I think one of the few academics I saw raise your hand related to decisions. Can you give me a sense of the decision that you're working on? Yes. A lot of my work is focused on land use decision making and the implications of land use decisions be for infectious disease transmission and anything like practiceable that land owners can do with their own property, private land owners to influence disease risk in their environment. OK. Awesome. And then are you working with stakeholders or other folks outside of the university? The Main Woodland Owners Association, which is a forest land owner organization. And I also work with a lot of social scientists on it. Awesome. A couple of my folks raised their hand that they work on more decision applications. Who are they again? Yeah, so Jen, can you describe a little bit about what you do? Yeah, not to ruin my exciting lighting top. It's OK. I work in the endangered species active listing programs. Pretty much all of our projects are all the research in our projects is geared towards answering the question whether or not we should list a species. OK. A couple of you all mentioned endangered species. There's a couple of you all that work on that. Is that right? Tony, you're another one. Does anyone have another application, Justin? Yeah, I work on primary malaria but disease intervention. So my research is on decision support for maximizing effectiveness of interventions for malaria. So you're directly thinking about whether or not something that someone will do will have an impact. And then I think I saw your hand. Mike, right? Yeah, so we're looking at decision support for mitigation and restoration habitat around the Great Lakes coastline or a migratory behavior and the migratory activity through the region. OK. Well, fabulous. I mean, I think you're getting a diversity. I mean, this is just a snapshot of some of the decisions that can be made. But I mean, similar to what my team works on, I mean, what you're hearing is that these are multi-stakeholder problems. They're really hard. There's science that's really important. And it's not always easy to understand how to best support those decisions. But I think one of the things that we're going to talk about is this links into forecasting. But it also is just links into how you can make smart decisions in your everyday life. I mean, honestly, I mean, there are tons of really hard decisions. And I have used this process to decide whether or not to take a tenure-track position at an R1 University versus taking a postdoc with one of my academic heroes. Spoiler, 10 years ago I took the postdoc. It was a good decision with a good outcome. And I've used this to think about how to make choices about what vacation I go on, whether to do big moves. I mean, there's how to buy a house and what house you want to buy given all the trade-offs of school districts and price and proximity of friends, commute time, all of these things that matter when you're making these choices. And as soon as you get beyond one or two objectives, it just gets really complicated really fast. So today, in the sessions that I have, they're not very math-focused because it's really hard to just think about how to do this and to structure it. It seems simple, but a lot of the thinking is just on getting the pieces right. The math of this is actually fairly easy. And just because I love books, I'll give you a sneak peek. So Mike has in his book a chapter on ecological forecasting. We'll refer to it a little bit. Two of my favorites of many. I have a whole bookshelf if you want more. Individual decision-making. So you're thinking about making smart choices for your own life. Links in the same process as how we think about it for environmental decisions. Environmental decisions are just even harder. Here's a great book that you can read literally on the airport ride home that will give you a big overview of things that we're talking about. Thinking about this within the context of environmental management decisions with multiple stakeholders and how science feeds in. Structured decision-making related to environmental management. All of the examples in here are about the environment. There's lots of other books out there, but this one, I think, breaks it down pretty well. And we're gonna talk about where forecasts feed into this. Mike set this up beautifully because, you know, we wouldn't really necessarily need forecasts if we could just have clairvoyance that would tell us exactly what would happen in the future. But forecasts end up being really critical to thinking about decisions for two reasons. One is we can't perfectly predict the future. So we need forecasts to help us do that. Part of the reason why we can't predict the future is that there's lots of choices that we, as people, make that impact the trajectory that we pursue. So if you think about climate projections, the largest uncertainty with those projections is what choices we make in terms of the greenhouse gases that we either emit or mitigate. It's not as much the uncertainties with the model, it's the uncertainties with the scenarios and what we choose to do that will affect our pathway in, you know, 2021. So thinking about decisions, I think it's really useful to break this down because we don't have just science-based decisions because inherently in any decision that we make, there's science that's part of it. But critical to a decision and it can't be divorced from a decision are our values. Either our personal values or stakeholder values. What do we care about? That's the whole reason why we're making decisions is we care about something and we want to maximize the likelihood that what we care about will happen. And then with especially environmental problems and sometimes with our personal decision opportunities as well, there's constraints that are imposed. Some of these might be regulatory, some of them might be a geographic, but there's constraints that have to be considered. So when we think about supporting decisions, there's a lot of pieces that go into it. You know, we've talked about the values, we've talked about models and projections, there's clearly uncertainty associated with it. There's information about the environment and what's possible responses that one might have. There's considerations like cost and revenue that end up being particularly important. When we talk about decision support, it includes two pieces. We have and we will talk about some of these tools such as scenarios and thinking about scenario planning, you know, data management and visualization, ecological forecast modeling, you know, integrate assessment models, things that feed into decisions that help to aid those decisions. But just as important as these tools are, are the processes to think about how we make decisions. So these are processes where scientists and stakeholders work together to understand these decisions and problems and how to scope the problem space and the decision opportunities that you have. It's thinking about co-production of knowledge. You know, ecological forecasters are one group of scientists that have a lot of knowledge, but if you're forecasting in Boston for something in Guam, it might be useful to also ground truth some of the things that you're doing with local knowledge of people who have lived there many generations. And so being able to think about how you build that local knowledge in, so part of what you're doing is understanding how to co-produce knowledge together so that you can represent that fully within the decision models and choices that you're making. Thinking about, you know, these ways of assessing the current state of knowledge and linking scientists and stakeholders together. There's lots of literature on how to do this well and far more examples of how to do this poorly. So we're not gonna get into a lot of these details, but just think about this when we talk about decision support. It's not just constructing a bunch of models that feed into a decision. It's both how we set that process up to interact with scientists and stakeholders, as well as how we build the scientific knowledge to actually make really smart decisions. And the whole idea of this is that, you know, if we do this well, we should have two things that result. One is, is you want good decision outcomes. The whole reason why you're doing this is you're trying to increase the likelihood of a good outcome. Can't guarantee a good outcome, but you're trying to increase the likelihood of good outcome. And that can, I hate this word consensus. I can't believe I didn't edit it out in the National Climate Assessment Report before it came out. But an understanding of the problem, the objectives, the options, and the scientific information that goes into it. So people being on the same page about how the decision is made so that then you have an understanding. It doesn't mean that everyone's, it's gonna be a win-win option, but it means that everyone understands how the outcome was, how the decision was made, and then the outcomes that result as a result of that decision. The other thing is, is if you do this well, you have process outcomes. And that means that you're building and strengthening relationships between scientists and stakeholders. You're getting buy-in for your decision so you don't run into roadblocks just because you've created this model or you've linked it with a decision model. It has output, you know, your preferred alternative. Great, we make that decision. And then you have a bunch of other people that are like, whoa, whoa, whoa, wait. Did you consider what I care about? I wasn't part of this process. So really making sure that we're thinking about both of these so that then you have outcomes that are achievable and good at the end. So during breaks and things, if you're interested in, you know, some of the more process element things, we can have some side conversations about how to set that up and manage it effectively. So I think it's really useful to then put some context around like what decision analysis and decision science is. So descriptive science, we're just gonna go out in the field and look at stuff and measure stuff. It's literally what do we see? What is it? And that goes all the way to normative science. How should we make decisions if we are rational actors? These are like process-based physical models. So these kinds of normatives of like, how should the world work given sort of a theoretical understanding. Experimental science, I think this is a bit, you know, where ecological forecast in. You're creating, you know, experiments about the world within a computer environment and then testing out whether or not those happen. What I work on is prescriptive science. And that means that I'm really focused on how can we make, improve the processes to make better decisions? So how can we develop better methods, you know, work more effectively with stakeholders, better integrate models to make better decisions? And so that means as we think about how we design and create ecological forecast models, you know, there's a couple of different ways we can do that. There's the sort of scientist perspective of we want to better understand the world and so we're gonna construct these forecasts and we're gonna push them out into the world. So this is, you know, not dissimilar from sort of, you know, the weather forecasting model. We develop really good forecasts and we push it out into the world and other people figure out how to use it. There's approaches where, you know, what we do is we work really closely with stakeholders. We understand, you know, the science gaps that they have and we design models or scientific information that directly feeds into what they need. What I'm gonna encourage you all to do as you all think about and work on your projects is actually a model where you think about both. So we think about the kind of science that needs to be developed to improve our scientific understanding, but we do that within the context of understanding what decisions may be influenced by that information and whether or not there's ways of designing the models and constructing the outputs such that it's relevant and useful for those decision contexts. In some cases, you may be working directly with, you know, a federal agency that wants to use that information. In other cases, the decision context might be a little further off, but if you're thinking about the decisions in addition to the scientific understanding, I think it creates an opportunity to do more. So part of the reason why I think, you know, this is just so incredibly important is because this just links perfectly into adaptive management because we have not fully realized this dream and this like philosophy of adaptive environmental management because it's really hard. But Holling in 1978 put in his book, when talking about adaptive management, it's concerned with how to deal with qualitative and quantitative data, how to use this knowledge of fundamental processes to construct models that can serve as laboratory worlds for the testing and evaluation of intrusions or interventions, developments, and policies. This is why models are so powerful in environmental decision making as we can't always monkey around with the world. And so we have to figure out a way of creating like an invisible world that we can screw up as many times as we want until we figure out the best way of making some of the changes that will lead to the outcomes that we want to see. I mean, how many times have we talked about introducing species into ecosystems only to have them become like invasive ginormous pests? Models, expert knowledge. I mean, these are things that can be brought into decisions so that we're not making poor choices because we're not considering all of the implications of what might happen. We're not scoping it too narrowly. So with that, I think, you know, one of the things that I was trained as an undergrad as an environmental scientist, but I really wanted to work on policy questions. And what I saw when looking at policy questions is that so many of the environmental decisions that I saw implemented, we lied heavily on models. So I thought, no, that's what I'll do. I'll do water quality modeling because I was super interested in fresh water systems and how to make them better. But my last year of undergrad, almost every person I talked to has a story like this, but here's mine. So last year of undergrad, I decided school is really fun and guess what, they'll pay you to go to school. That's really awesome. No one in my family has ever gotten a PhD, but this sounds like an amazing deal because it's the most money I've ever made in my life. Why don't I go around and see if some of the people who I wanna work with are doing really cool things in the next five years and wanna take me on board as student. So I went around and I basically visited Duke. Okay, so my decision process for my PhD is not one I recommend to any of them because Yale was my backup and I only applied to two schools. So I went to Duke and I met with, who became my advisor, Ken Rekow. He works on water quality and water quality modeling, but with a policy emphasis. I met with a couple of other people and one of the people I met with just said, everything that you're describing is focused on decision analysis. Have you ever heard of this? And I was like, no? And they were like, read this book and it was a book written by Ralph Keene, K-E-E-N-E-Y, no relation, but I'm sure we have some Irish roots somewhere. We just changed the spelling. And he wrote this book called Value Focus Thinking. So I ordered it and spent the weekend reading it, like I don't know, most undergrads. And by the time I got to about, I think chapter five is where the math started. And I was like, I need to figure out how to do this because everything I'm reading here is a formal way to describe how I think about the world and the decisions and how we can structure making good decisions. So I say that because I think there's two things with this. It's like I said in a lot of the sections that we're gonna deal with, it's not as focused on the math because I think the thinking about how to structure this is really hard. And I think it's useful to kind of go through that. It's also because I view math as a means to an end, not the end of itself. And so I don't learn through equations. I really learn through context and decisions and applications. And then the math is more just a way of getting there. Is there other people who kind of think that way? Okay, so I have a couple of fellow compatriots in the room. We're gonna get along well. But I think the thing is, is that this whole thing also applies to ecological models. So we're gonna think about it within various decisions. But you all are making decisions throughout this week and you're making decisions about how to structure your model, what's included, what's excluded, how you think about the real world and how you're abstracting it for models. And so you're gonna be making a lot of those kinds of decisions as you go forth. We're just gonna break down like a rigorous process to think about how to do that better. We can talk during some of the project time, if it's useful, about how you can incorporate that in terms of like modeling and thinking about that. So yes, decision analysis, magic. It was really great when I was at Duke because part of the beauty of that was they had phenomenal environmental folks. I got to work with people in Jim's Clark's lab and met my QTA, my Bayesian statistics class. But I was able to spend half my time in the business school with a decision scientist who found environmental problems a really neat application. So it all depends on your perspective of what's novel. But that gives us some sense of what's interesting. I think the one other thing that I want to emphasize before I move to the process is we make a real strong distinction between decisions and outcomes. So I mentioned this earlier. What we're trying to do is make really good decisions that we think will lead to outcomes. But what outcomes occur are based off of the choices that we make. They're not, they are only influenced by those choices. Once I learned that, I can't tell you how freeing it was because it meant that when I was making really hard choices about like what home to buy, what jobs to take, all of these big things in life, if I had thought rigorously and made the best decision I could based off of the information that I had right now, then if it led to a bad outcome, there's nothing I could have done to influence that because I had gone through all of the steps that I thought I could do to lead to the outcome that we wanted. And I think that's something we should keep in mind as we think about environmental decisions because there are even more complex and it's really tricky and there's a lot that we can do to make better decisions but we're still gonna have bad outcomes and that's an opportunity to sort of learn and adapt. And when we think about our processes, that's something that we can do. A lot of the diagrams that I'm using are coming out of structured environmental decision-making largely because I want you to be able to use that as a reference if you wanna come back to this because you find it as awesome as I do. And so what we're gonna do is we're gonna focus on this method called pro-act. Pro-act is basically the process that we're gonna go through. It stands for problem, objectives, objectives, alternatives, consequences, and trade-offs. Problem, objectives, alternatives, consequences, and trade-offs. So then when you map that, we're gonna look at two conceptual models because they link to structured environmental decision-making and ecological forecasting. The way that they represent problem, clarifying the decision context, really understanding what your decision choice, what your decision opportunity. Objectives, defining objectives and measures. So this is getting to both what do we care about and how do we measure what we care about. So objectives also includes performance, measures, indicators, attributes, metrics, criteria, okay. These are all words more or less indicate the same that are pointing to the exact same thing that we're gonna be talking about. So you'll see these words, criteria, metrics, attributes, indicators, performance, measures. If we use those words, they're more or less the same thing but you'll see it in structured environmental decision-making and the ecological forecasting book as performance measures. Developing alternatives, here's our choices. Estimating the consequences, here's where ecological forecasting comes in. You know, really understanding what's gonna happen, whether it's predictions or projections. Evaluating your trade-offs and selecting. So how do you compare your different choices? And then really thinking about this within an adaptive management context because with environmental management decisions, like guarantee we're probably not gonna get it right the first time. So, and even if we do, we wanna show that we did. So developing processes to monitor review. This is also where interactive ecological forecast come in because as we're collecting new data, we should be updating our forecast and understanding both how to improve our forecast but also how that feeds into these decision processes. Mike in his book has similar layout. I'll highlight two things that are different but they map here as well. So generate alternatives and scenarios. So the reason why scenarios are brought in here is if you're thinking about ecological forecast, the boundary conditions matter. So you can't think about the consequences if you're not controlling for the conditions that you're trying to set up. So the land use scenarios, policy interventions, you have to be able to be comparing similar conditions in order to really make smart trade-offs. And then forecasting consequences. Consequences and decision models involve forecasts but there are a lot of objectives that are outside of the domain of ecological forecast. There are objectives related to aesthetics and recreation with our national parks. There are objectives related to cultural values associated with a lot of our lands that are on current or former indigenous locations. All of those things matter and there's space for them in decision models. It's just one of those things where you can't just forecast something and expect it to give you the answer because it's one piece of this larger decision that we're making and there's a lot of objectives that we need to care about when thinking about those choices. And then evaluating trade-offs. I would say with this, there are but not represent this diagram, these loops relate to adaptive management and intratip forecasting. So just keep that in mind, it looks linear but it's really everything's a giant circle. So I thought it would be useful to actually go through an example. We're gonna think about the decision of a flight to ESA and breaking down how we make that decision. If you're not going to ESA, any kind of professional meeting, let's get a sense from everyone about what you think about when making a decision about a flight. Jeff, can I start with you? Sure, I think usually first about cost. Okay, great. Cost. Ian, is that right? Yeah, how much per week you get from the PI or how much you spend on it? Okay, so cost plus like perception of other people. Like grief from the PI. Rushi, what do you think about? What time I'm leaving and arriving? Mm-hmm, same amount of time. Is it Jenny? If there's a lot of layover, just length of the layover. So if it's not a direct flight, whether or not there's layovers or in the time? It's where you need to be. Yeah, Whitney? Okay. I broke cost up into a couple categories because these days you often have luggage fees in addition to your airline ticket. Yep. And sometimes you have multiple choices of airports like Washington DC area. So you might consider your destination and how that relates to public transportation or Ubering or additional costs to get to your hotel. Excellent, so additional fees like baggage fees as well as if you have a choice of airports you know, almost like how convenient it is to wear your meeting and the like. Leland. I consider who's paying for it. So as a grad student, I would always book the cheapest and with the most horrible layovers possible generally. This was saving cost, saving money, but now that I have some grant funding I can spend a little bit more on this than before. Okay, so who's paying for it? Which I think can be an important factor. Alice, thank you. Maybe pre-event flight decision is driving option. Like how far am I? What's the mental card price? Yeah, so we're gonna come back to that Alice because- That's an alternative. No, you've reframed the decision. What you've done is you've changed it from a decision about what flight do I buy to how do I travel to this meeting? Which may create both some additional options but may also introduce some additional objectives. So we'll come back to that because I wanna actually build that out but for right now we'll focus on the flight because the travel adds an additional layer of complexity. Naomi. Yeah. Okay. Sometimes I think about the airline. Okay. I think I might be stuck flying spirit air. Yeah. I've been waiting so long to buy my flight which is really suboptimal. And for what reason is that suboptimal? Because- Because there's a worst. So it's fees for everything. Have you ever flown spirit air on? No, no, and there's many reasons I have not- You don't know why it's suboptimal if you've had. Right, so lots of fees. I think I heard something about- The seats are super small. Okay, so there's a factor of like, yeah, uncomfortable. You're laptop because the tray table is like this tray. So you can't like do any work on the plane. Okay, yep. I'm usually doing it on the way to a conference because usually I'm essentially on top. No judgment. Okay, okay, so yes. Whether or not this space will actually be comfortable to do things. Yeah, Justin. Kind of related to that. What do I need to bring? So do I need to bring a poster? Do I need to check a bag? Mm-hmm, yeah, so what do I need to bring and then what's associated with bringing that stuff? Other things, yeah, Winslow. For ESA specifically, I think about what is the probability that I'll get scheduled for a Friday morning talk? That's a weird half day session. Okay, yeah, yeah, so which I'm guessing you're trying to buy flights before the program is even put out. So there's an additional layer of uncertainty. You might be able to buy a cheaper flight, but you... I think home at a better hour, you know, when you get home at like midnight, but almost 6 to 8 minutes instead. Yep, okay. So considerations of the meeting and the meeting time and uncertainty associated with that. Other things? Yeah, Tony. So in addition to the airline, do you wanna collect miles for a certain airline? Yeah, and that can be, you know, particularly important for some people. So, you know, miles or loyalty and implications of that. Yeah, Mike. So if you have to be somewhere at a particular time or home at a particular time, you have to sort of backtrack which flights are not working that context. Right, so you're thinking almost of the constraints that end up happening as a result of... Yeah, so like, you know, if you have to take a bus two hours into the city or something like that, you have to sort of figure out where the bus is and then you have the last time you can get ahead of the center. Yeah, yeah, so figuring out, yeah, all of the additional like linked decisions that end up happening as a result of, you know, a choice of a flight. Zoe, right? Yeah, I'm thinking how jet lag would affect your sleep schedule and for a red-eye flight overnight or does a time zone change? Yeah, so not just the amount of time that you're traveling but, you know, the time that you would, you know, arrive and how that then affects your ability to be productive during that time. Other, yeah, Whitney. I'm often traveling with other people as well. If I have lab mates or my husband going to the same conference then it's almost another can of worms where it's my preference for each of these decisions but it's also upstairs as well. Yeah, yeah, and you can think about that in two ways. You can think about that as a constraint. So, you know, what's gonna be feasible for other people and how does that, you know, constrain what's possible but you can also think about that as, you know, almost like a couple of individual decisions that then you're trying to develop some consensus and negotiate what's the best choice for everyone where it might not be optimal for any one person. So, yeah, you basically created like a giant complexity in terms of how we would structure a decision model in that context. I love ecological forecasters. They always think about like these much more complicated situations. Tony. This may be more relevant for me but whether or not you're flying a domestic or international airline. Mmm. You're constrained by. Yes. Fly America. Yeah, we have that with our grants as well. When you route through Japan, you're limited by your option. Yeah. Any other things people wanna add to this list? So, one thing I actually think a lot about is the ethics of the different airlines. Ever since United pulled that person off the airline, I've avoided them. Okay. Sorry, I try to stick to that. Okay, ethics of the airline. And Jake, I think you add another one. Yeah, I think about legroom a lot. You know, funny, that doesn't factor into my decision too much. Yeah. Sure, flight, and I won't care if my food is long, but okay, that's true. You can catch what these notes are. Yes, well done. Well done, Chris. So, this is only a decision about a flight. If you travel a lot, you don't necessarily go through and like brainstorm all the things that you might want to consider. You probably have developed some mental shortcuts around, you know, these choices, or potentially might be constrained. You might have narrowed like the decision context a little bit so that certain factors may not come into play. But this kind of, layoff sink, I love the commentary. Take this and actually think about how they link to objectives because this is brainstorming and what you see here are a lot of things that matter, but they're more means to what we fundamentally care about. So let me take a couple of colors and let's circle a few things. Okay, so let's do the easy ones. Okay, I see things about cost. Okay, we see things about the cost of the flight. We see this about the sort of grief about the boss, but we're gonna keep that a little bit separate. There was one about baggage fees around here. Actually, Chris, can I ask you to come up and circle where these are? Because I think you're gonna be able to find them faster than me. Yeah, someone mentioned cost again and they were like, do you still see the cost besides the flights that was kind of there? Yeah. Okay, so one of the fundamental objectives that I'm seeing, and it might show up in other places, but how do we minimize cost? And that can be the cost of the flight, but it can also be associated with baggage fees. You might also think about it also in terms of partially related to airline miles because you can actually calculate basically the monetary worth of an airline mile and that could feed into your linkage to cost. So if it's an airline that you fly frequently and you'd actually get value out of that versus an airline you don't fly frequently, you can actually calculate that. Sure, yeah, we'll give it a half circle because I think there's another piece that goes into that. Okay, so minimizing cost as one of your key objectives. What do you all see as another objective? Comfort and convenience, excellent. So that gets into this airline quality, the small seats, the leg room, how horrible the airline is. I think ethics is different. Yeah, leg room, there we go. Another sort of fundamental objective that you see coming out of this list, time. Yes, time. And yes, layovers, the amount of time that the flight takes, whether or not there's additional time that's added based off of the airport that you choose to go into. So the amount of time that it takes. Yeah, I think jet-like fits into that, but that might also be something where we wanna think about then how we consider that and measure that. I think that's a really important one. So let's think about it in terms of time, but we might actually pull it out, relate to something else. Other objectives that you're seeing in here? Maybe like maximizing your productivity or maximizing what you get out of the conference. Okay, and what sense for the choice of an airline flight? Well, like the jet-lag thing. Okay. I guess I was thinking about how you're coordinating with your coworkers or deciding flights before knowing session day and time. Okay, excellent. So maximizing productivity during the conference? Yeah. Or just like what do you get out of the conference? Other objectives that you see? Maybe like considerations involving other people on some passion. Yeah, so I think that was linked into the productivity, but I think you're thinking about it slightly differently. Ali, can you give me a little bit? The ones involving like the boss and the boss's perception as well. Yeah, so almost like, so you can frame it as a constraint. You can also frame it as like an objective of like almost like minimizing social irritation. We can wordsmith these later, but yes, something about like minimizing social irritation, which I think is a good one because if you think you're gonna get grief from the boss about certain things or would the fly America go into that as well? Kind of in the theme of making the higher ups happy or following rules that are set from above? I would consider that more of a constraint, Whitney. And the reason is, is that it limits the number of options that you can pursue. Yeah. And I think there's one other one that relates to Winslow's comment. Yes, the ethics. Yeah, so maximizing sort of ethical considerations or exactly how they get worded. Usually you come back and wordsmith. So let's actually erase this. Can I ask for, thank you, Chris. Can I ask for another repertoire for this? Okay, so let's actually take this brainstorming and let's build out our objectives. So let's actually list our objectives. This is what we're trying to create. This isn't the only way that you can construct things, but it's super useful. So what we do is we list our objectives. So these aren't things like leg room. Instead, it's things like maximizing comfort. We have our performance measures, which we're gonna go through. We have the units and where we might find the information. We also sometimes indicate like the magnitude if we haven't included in the objective or if our performance measure is different. So you were trying to maximize leg room, but the way that you measure it may have like a different relationship. And then various alternatives up here. Okay, so we're gonna start with our objectives. So we had minimize time, maximize comfort, minimize social irritation, and then maximize ethical behavior. Okay, so these are what we call fundamental objectives. These are things that we care about because it is fundamental to our decision. Now, how we get to that, part of what we've talked about are means. And this gives us a sense of potentially some of the performance measures we would wanna use in order to evaluate those different objectives. For minimize costs, what are some of the ways that we could measure whether or not time, sorry, cost has been minimized? Dollar spent, and dollar spent for what? From door to door, literally like walking out of your house, all the costs you would incur, putting in the plane, baggage fees, and then coming back. So you would wanna include the flight costs, the drive or taxi cost, parking, baggage fees, other fees that Spirit Airlines adds on to things because they're really annoying. But basically your door to door expenses. And when you're doing this, sometimes it's helpful to just make sure like you're listing all of those things just so that if someone else were to look at your decision model, they would understand all of the factors that you're considering. It's not just your flight costs, it's also the door to door costs. Minimizing time, how would you wanna measure that? Yeah, Jenny. I guess how many layers you had or if, yeah, you could avoid just having a red flight. Yeah, at what time you had a week out, two cuts your flight, sometimes two. Okay, so I'm hearing things related to the total time that you're spending flying. And then I'm hearing something related to the, yeah, like the convenience. Cause I think you're hearing things with comfort, comfort related to me personally, with the flight times that might constrain the options that I consider. And considerations once I get onto the plane. We had a bunch of ideas around maximizing comfort that were highlighted, leg room, comfort of seats, size of the dropdown table. Yeah, the tray, things related to sort of the service that you get from an airline. Yeah, Ian. So in some of these, they're potentially qualitative. We're gonna talk about how we deal with qualitative and quantitative measures. Yeah, so maximizing comfort, I think this is what you're getting at is, there's things that you could do to like quantitatively measure each of those. Like inches of seat or something, but that isn't maybe how we think about, and just some of the more inherently qualitative. Yeah, so let's actually use comfort as a way of doing a constructed scale. So we think about multiple things with comfort. Some of those we can measure. Some of them we qualitatively assess. But what we could do is we could do a constructed scale of like, you know, spirit airlines to like, you know. Window, middle, aisle, seat, like that's not easily quantified. Right. So maybe something like that. Yeah, Zoe. I feel like the comfort during the flight might be a separate scale of like the disturbance that causes in your life before and after the flight. Like from the waking up early sort of thing. Okay, so I think what you're then highlighting is that there might be two indicators or performance measures that we might want to use. So one of them might be sort of a disturbance indicator. You know, how much it disturbs our life. You know, both in terms of traveling, but also once you get there, like whether or not you have jet lag or things of the sort. And then the second is a comfort, like once you're on the flight or that flight experience, how comfortable it is. With this, there are certainly things that one could quantify and you could list out lots of different types of indicators and be fully comprehensive. But for this example, let's just think about it as a qualitative scale because I think most of you all, when you were talking about this, you were like, well, I kind of know with certain airlines how comfortable it's gonna be. Emirates Air, it's gonna be like pretty luxurious. Spirit Airlines is gonna be pretty hideous. So we can think about that in terms of a qualitative scale of like maybe a one to five ranking of like one being the worst and five being the best. You could also think about that with the disturbance indicator. So whether or not the flight is gonna have a negative impact on all of these other factors that you care about. Yeah, Naomi. I feel like I could put a dollar value on those things by like some sort of willingness to pay. Well, because I have like, I don't necessarily have like clearly articulated, but when I'm looking at flights, it's like, okay, well, Delta costs $200 more than Spirit. I won't pay that much more on Delta, but if it's $50 more, I would pay to go to the top Delta. So like you kind of put a dollar on those comfort things of like what premium will you pay for them? Yeah, which is, I think Naomi, you're jumping ahead to, how do we think about trade-offs? So, and that's not a bad thing. Some of it you can do through dollar values, but some of it you can also just do through the natural measures or the proxies that you're using, but you're basically saying, okay, I know how to trade off costs and comfort. Yeah, well, more so than I know how to say, like break these things from one to five. Like the, maybe it's just because we go straight to the trade-offs when we're thinking about these things. Yeah. And thinking about it in this way of like one to five is maybe less natural. Well, and I think here's where it comes into environmental decision-making. Okay, so we can jump ahead and think about like all of the alternatives and how we would make these trade-offs, but that's really easy to do if it's me. I know my preferences and values really well, but I'm sitting in a room with 20 other people who have slightly different, we might agree on these broad objectives. This might be something that we kind of agree. We might be able to create agreement on how we would measure it. There's choices that we have, but the trade-off comes in is the way that you would trade-off cost versus comfort is not the same way I would trade-off cost versus comfort. And so that process of how you do that, there's an element of like wanting to be transparent and rigorous of that process, but there's also an element of, let me, yeah, let me just stop there. I think it's really important, especially for environmental decisions with multiple people involved and a lot of scientific information to be really clear about how you're developing the science to make these assessments, but also how you're making those trade-offs because you may choose to fly a Spirit Airlines because you're like, oh, it's hugely uncomfortable, but it costs me, you know, it's $200 less, so that's worth it to me. Whereas I would be like, no. That's absolutely fine, but what you would want is, you know, if you're constructing this and multiple people are using it, you know, it may not be, it's not necessarily important when we think about how am I gonna make a personal decision to fly to ESA. It's really important when we think about how do we make choices about how we manage our environment because what we wanna do is keep each of these steps transparent and separate. We wanna be able to be really clear about these fundamental objectives that we care about. We wanna be really clear about how we're gonna measure those. Not calculating what those are and not making assignments, but what's the best way to measure the things that we care about. We wanna think really clearly about, you know, what our choices are. And one of the reasons why we don't start with our choices is for two things. One is this is where advocacy in a bad way comes into play of like, I have predetermined what I want to happen and I will find facts and ways of justifying the choice I have already made. When you think about your goals and objectives, it creates an opportunity to expand the way that you think about your alternatives. We see that actually in the example that that Alice pointed out of like, well, I might not fly. Could I drive instead? You know, that might be a more environmentally sustainable option, which might be another objective that I have when making a choice about how to travel to a meeting. And that expands my options set of like, both driving, it might also include options like trains and other things. So that's a lot of the reason why we go through this process is not because for these kinds of things it necessarily helps us to make a better decision, but it helps to break things down so that then when we're dealing with even more complex things with multiple people in the room and a lot of really uncertain science and we have people coming at this and calculating consequences in different ways, we can point out where the fallacies are in the decision making process. Is it because we're having conversations about fundamental values and we're talking past each other because we haven't actually had the conversation about the things that we care about? Is it because we're thinking about what to measure and we're thinking about what to measure in really different ways. And so we're not actually able to compare and talk about things. Are we bringing different science barrier or different boundary conditions or other factors that aren't gonna lead us to think about these things concretely? You know, are we thinking about the full range of all alternatives that can occur? So this is part of the reason why we wanna, why we wanna break it down like this and not jump to making the trade-offs. So clearly these are important. We're just gonna assume that they're basically the same. So we're gonna give them the same number. So they don't end up factoring in. So let's look at three alternatives. Flight one is gonna be $600. Flight two is gonna be $300. So we know what Naomi's gonna pick. And flight three is gonna be $650. In terms of minimizing time, this is gonna be total flying time. It's not gonna be total door-to-door time. But what we're gonna assume is that the door-to-door time is the same so that if we just calculate total flying time, it's a perfect proxy. It's a perfect substitute for thinking about total time. So this is three hours. This is six hours and this is 10 hours. And then maximizing comfort. So we're gonna do this on a constructed scale. So one is the worst and five is the best. This is four, this is four. And this is a new airline that we don't really know and we've never flown before and none of our friends have flown. So we literally have no information. So all we can say is that it's somewhere between one and five because I really don't know. Giant question mark, giant question mark. Let's just focus about it as the during flight experience. This disturbance indicator, really important. We're just gonna put this as one we're not gonna consider right now or we're gonna assume it's exactly the same. And that's just so that we don't get too far down the rabbit hole before talking about environmental examples. So with this, we have three options. We have three alternatives. I find it super easy and this is actually something they do in the military. We're just gonna put a green around the best in all cases because it's not always the lowest or highest number. These are kind of the same. And then the worst, this, this. And I have no idea what that is. So what we can do from this is let's first eliminate objectives that don't matter because we have figured out some of these are more or less the same. Okay, so we're not talking about minimizing social interaction because we've given that the same score and ethical behavior is the same even though we know it's not. And similarly, we're not gonna think about disturbance. That said, here, what we see is that these two are exactly the same and this one we don't really know much about. So I would say for this one, we can't really make an assessment of differentiation of levels of comfort between the airlines. So we can eliminate that objective from consideration. Now we have three alternatives and two objectives. What we can see is this is the worst on all of them. So this one's dominated. That means there's no way that this would ever be the best choice so we can eliminate it immediately. Jeff. I did a question about you just lined up and maximized comfort, but it seems like you could potentially say, well, how risk-averse are you? I'm really feeling fast and loose at my time. Yes, so this gets into, in this case we've simplified the problem and created it sort of as a discreet entity, but yes, if you're risk-averse and you're like, hey, it's maybe the new Emirates. Sure, let's try it out. Maybe comfort is really important to me. No one knows anything about it and I can learn a lot. So absolutely, there are cases where that level of uncertainty is not necessarily a bad thing because being able to have an experience where you would have a comfort level of five is worth it. We're gonna say for a group like this, we're not gonna consider that, but you're absolutely right. It absolutely depends on how risk-averse you are and what you see in terms of the value of learning. So once you have a set of, basically you've constrained your set of options because you've eliminated objectives that don't matter or you've eliminated options that are dominated. In this case, we are risk-averse. We're not risk-seeking. We are gonna eliminate this option from consideration. This comes down to a trade-off between cost and time. So given your own values and whether or not you typically have to pay for your travel or it's coming off as something else, how many of you would take the $300 flight that would be six hours? How many of you would take the $600 flight that would be three hours? And so yes, what you're seeing here is that people would make different choices and let me give you one of the cases where I actually would consider another objective when making this choice. So if this is coming out of my personal funds and I do not wanna spend a huge amount of expendable income on professional activities, I would probably choose this cheaper flight. However, if I'm doing consulting and the consulting company pays for my travel time and my hourly rate is $200 an hour, this is a better option because it's cheaper overall for that company. So you can then think about almost how much your own time is worth in certain situations which may be different depending on who pays and the context of how they're paying to factor into how you would make these choices. So this is breaking things down in a particular way. The thing that's really powerful about this is it separates the scientific pieces from the values and it recognizes because we're making decisions of values are equally important. You can't make decisions without them. It also means that the science that you're developing, the information you're developing directly relates to the objectives that are relevant in a decision. Doesn't mean that if you're developing a forecast and someone's going through this kind of rigorous process that they wouldn't pull that forecast in. But if you're designing specifically for a decision, you're pulling in information that fits in and helps you to understand the likelihood of being able to achieve that objective. So we talked about a few things because Alice brought up this question of travel and that can bring up additional objectives. So we talked about almost environmental sustainability. So you could do that related to greenhouse gas and the equivalent greenhouse gas emissions that would occur as a result of that travel choice. Yeah, so I think with this, with the decision framing, if you're expanding it, part of what you're oftentimes doing is potentially adding more objectives that you care about and increasing the number of alternatives that you would consider. So you're considering alternatives you wouldn't. Naomi. The consequences step, is that the like $600 versus 300 and the three versus six, those are the consequences. Like what are the measurable outcomes of taking alternative one versus alternative two? Absolutely. That's the like forecast. Right. If it's not, it's just how much the flight is. Right, I mean, in this very... Think of the odds of a delay. Absolutely, I mean, you can instruct these probabilistically because it's also like maybe you're buying your flight within a week and so you're trying to get a sense of like what flight you would wanna buy. There's uncertainty about whether prices would jump or potentially get lower. And you can look at the volatility in prices by looking at kayak or some other sites like that. And you can look at the time because, flying through Chicago in winter. Yeah, exactly, exactly. So you also consider those aspects because it might be three hours, but if it's January and you're flying through Chicago, you might wanna add some uncertainty bounds on that because that might be a minimum time and it might actually be a couple of days before you get out. The Chicago airport is not very comfortable. So yes, yes, we're looking at it deterministically, but yes, you can absolutely look at it probabilistically and through that lens. Chris. In this example, it's kind of well down to this simple trade-off between two things. How heuristically or algorithmically do you kinda do these when you have like more than two things on the table? Cause my brain starts to freeze up if you change those ones on the bottom to different numbers. Yes, dealing with more than two things is super difficult. It wasn't a problem that was solved until 50 years ago. And so the short answer, and we'll actually talk about this more on Friday, is what you end up doing is using something called an index of desirability. Also talk about this in terms of utility. And what you're trying to do is normalize these different objectives on a scale of like zero to one or zero to 100, whatever some kind of normalized scale so that then you can say, okay, let's see, 300, 650. This is the most desirable. So this is gonna be one, least. So with this, 300 is the most desirable, 650 is the least. The question comes in of whether or not a change from a dollar of 300 to $301 is the same as a change from 649 to 650. So is this linear or does this have a different shape to it? So it's more than just a simple normalization. You're actually, it's like the step can, yeah, there's this function as you go from course to lease. Yeah, and so with money, with companies and large decisions, they should have a linear index of desirability. A dollar here and there should not matter when you're dealing with billions of dollars of like corporate revenue. It may matter a lot for our own personal finances and decisions. And then similarly, so comfort is on this qualitative scale. And I think Ian, you were pointing out qualitative scales are really important, but the difference between one and two may not be the same as the difference between four and five. So you might think like in terms of comfort, being able to get from a one to two is like a big jump. This is really, this is like me having my knees up in my like nose versus being able to have them against the seat. So that ends up being a big jump, but from a four to five, not so much. So, it might have this function where it's, you get a lot of gain from going from one to two, but four to five, pretty minimal, and being able, this comes into play when you make trade-offs because what you will eventually wanna do is be able to normalize based off of a utility or an index of desirability. And this is not the same for everyone. So, for this jump in comfort, it might matter a lot to Jake because he's really tall, I'm not so much. And so, this is likely to look different from me for a one to two, I probably am still not touching the seat in front of me. The thing about this is that you're constraining and based off of like the best and worst case scenarios, of values for this, it's not infinite, it is constrained. And this is all based off of preferences. This is not scientifically determined. And so, when you get into trade-off models, there are rigorous scientific processes that you can use to construct and make those choices, but if any decision analyst tells you it's a scientific determination, they're fooling themselves because this is all, like the way that you translate these values is based off of what you care about. And so, we talked about this in terms of like even swap. So, this is what you're talking about of like, I'd be able to calculate my willingness to pay and an additional three hours is maybe $100. So, if I'm looking at this, this is a much better choice. So, I can do what they call an even swap. So, I can make a calculation and determine what would be the equivalent. This is a more sophisticated way of doing it that ends up being really important when we're working with environmental questions. Let's go through a couple of examples to make this concrete for environmental decisions. All of these are coming from that structured decision-making book. So, we talked about this as fundamental versus means. So, think about leg room and tray tables as being aspects of comfort, but ends objectives or fundamental objectives are what we ultimately care about. So, here's an example related to sort of fishing experience in lakes. So, what we're trying to do is one of the things we care about is maximizing fishing experience. So, the way most ecologists might think about this is we wanna increase the numbers of fish or increase lake productivity or increase lake nutrients or fertilize the lake. So, you might hear all of these things. But what you wanna ask is if our ideas, what we wanna do is maximizing fishing experience? The question is, is how do we achieve that? What are the pieces that matter? If we're starting here, because this is easier to think about is like all the different things, the question is, is why is that important? And when you stop is when someone responds because it is, it just is. So, increasing the number of fish, the reason why we care about it is we wanna maximize fishing experience and why do we care about it? Because it is important. It's just fundamentally important to what we care about. This ends up being really important because what we wanna do is focus our decisions on these end objectives while realizing these help us to construct our performance measures that help us to understand whether or not we're gonna achieve that end. But if we think about it only in terms of lake productivity, we're not necessarily including or measuring some of the things that we care about and we might constrain our decision too narrowly. Okay, so characteristics of a good objective and I think we got there a lot with our flight example is so we want it complete. We wanna include everything that matters. We want concise. So we don't want anything that's unnecessary or redundant. So things that would be outside of the scope of what we're really considering with this particular decision. Such as like, I don't know how much fun we're gonna have at the conference because we're focused on decision about how we get there. Controlable. So these are things where there is some choice to be made. So you have the ability to make a decision about the level of comfort you would have on a flight. The airline has set parameters but you have a choice about which airlines you apply. And the other thing is that it's affected by the alternatives that you're considering. So this is why we eliminated some of these objectives. So we thought all about this but given the options that we were looking at some of these objectives ended up not mattering. So we can eliminate them so that they were only focusing on the ones that we could actually trade off and make choices about. Understandable. You would want this to be both understood by you but also someone else and especially when you're working in a multi stakeholder context. And then preferentially independent. This means that I don't need to know what's happening with cost in order to make a determination about time. These are independent of each other. We can assess these. These do not depend on each other. Performance measures and indicators. So these are critically important and this helps you to understand what matters in the decisions and maybe some of the outcome variables that you would want to consider in your ecological forecast. So thinking again about the decisions that one makes and what people care about in those decisions may give you a sense of I'm looking at phenological changes but being able to predict the start of spring ends up being really important for certain types of decisions and being able to do that on the lifetime effects agricultural planting and whether or not we are able to hit our cherry blossom festival on time and other things that end up being important. So you can think about the performance measures that one would think about in terms of the ecological forecast. So let's go through some examples. Objective, we're gonna minimize negative health effects. So that's a fundamental objective that we're trying to do. So there's three ways we can think about that. So one of the ways is you can think about that in terms of fatalities per year. So just the number of people. The example two, this is where we got with some of this where we were building out and adding complexity. So one could think about it in terms of fatalities per year, reduce activity days in the average population, reduce activity days among sensitive populations, hospital visits, respiratory events, cardiovascular events, asthma events. So you could build this out in a much more complex way. And then you could also build it out but just not quite as detailed. So fatalities per year and then reduce activity days in the average population. So what they've done is they've collapsed all of these activity days and these aspects and then related thought about it in a more aggregate sense. So three different valid ways of assessing the objective of minimizing health effects. Let's look at another example. Candidate performance measures for maximizing water quality. One of my favorite issues. You can think about it in terms of concentration of dioxin and use the original units and then the cost of making those improvements. You can also think about it in terms of number of exceedances. So this comes into play when we have environmental regulations and you have some cutoff. You can think about it whether or not exceed some kind of environmental standard. And then you can divide it based off of rate payers and then calculate how much it is per household which may impact a decision slightly differently than if you're thinking about total costs. This really depends on the decision, the decision context because the performance measures you choose matter because whether you're thinking about it in terms of the average cost to an individual versus like a municipality is really different. And people because of mental shortcuts they use and things that we know about how the framing is constructed they may have different preferences based off of how you construct those. So natural versus proxy. So for environmental considerations oftentimes with ecosystem services or like think about structure and function and let's monetize it because then it's included in decisions which is true if you're doing cost benefit analysis and that's the way that you're making trade-offs. We can make trade-offs in slightly more sophisticated ways and cost benefit analysis, same kind of general concept. We just don't have to monetize everything. So here's three objectives. Let me read these. I think it's hard to see in the back. Maximizing population of boreal caribou. A natural measure. This is something that it like naturally measures that specific objective such as minimize costs, use costs. This is abundance or the number of caribou. That's really hard to do. So oftentimes we might use a proxy like the area of habitat. Now when you use a proxy as Mike pointed out earlier in his forecasting there's uncertainty around that proxy. You know it's not a perfect measure of what we're trying to achieve and you would want to account for that when making those considerations. Minimizing air quality. So this might be the number of respiratory cases but you could also look at this as a proxy in terms of emissions of particulate. So different ways of getting at the same objective and then minimize impacts on Aboriginal culture. This is getting at some of those social factors that are critically important. So here there's no obvious natural measure. This is to your point, Ian, of like how, what? How do you do that? A proxy might be like number of culturally significant sites that are affected. So that might be a suitable proxy. It might not be a proxy that one of your stakeholder groups accepts. And so you might need to move to something like a constructed scale. So for example, you could do that for the Aboriginal cultures but here's an example of the way they've constructed a scale for public support of a facility siding. So this is whether or not there's support for siding like hydropower. And so what they've done is they've constructed scale 1, 0, negative 1, negative 2, negative 3. One is their support. No groups are opposed. That's awesome. Neutrality, groups are like indifferent or uninterested. You might have something that's kind of in between 1 and 0 but you have that. One is there's controversy. Two is there's action oriented oppositions if you're working against you. And then three is strong action oriented opposition of like two or more groups. So there's a way of then classifying this and then being able to include this in the decision process. Another environmental example is think about eutrophication status. Eutrophication manifests itself differently in different systems. And so you can describe the characteristics of like an illegal trophic versus like a hyper eutrophic lake but the characteristics of how you would map to that kind of categorical scale. It's not as easy as just like taking nitrogen phosphorous, algae and secchi depth and just being like perfect. This lake is like mapped as eutrophic. It's a constructed scale where there's not like a perfect way of perfectly measuring it. Does that make sense? So three different ways you can think about constructing performance measures. What you want to do is make sure that you're not just taking data or model output and trying to gerry-rig it into an objective. You're really thinking about what will be some of the best ways of measuring this. Are there ways of doing that? Or are there ways of configuring models and other things that we have so that fits up? Alternatives. So we talked about why it's so important to start with objectives and performance measures. And the reason is this theory called value-focused thinking that has been empirically shown of if you start with what you care about, you will expand the number of options and alternatives that you consider. So think about the ESA example. If you reframe that as like a decision about professional development, ESA as a conference might be one way that you choose to get that kind of professional development experience. But there might be other mechanisms. And there may be particular goals that you have about getting new scientific information, potentially meeting with new colleagues, being able to generate new funding sources as a result of it that then allow you to do new science. So there may be means and fundamental objectives that you care about that then may broaden the scope of your decision and broaden the kind of options that you would consider to determine whether or not taking a week out of a year would be worthwhile to go to the conference or whether it would be better to, I don't know, take an ecological forecasting class or go on vacation or other things. And then finally, consequences are where ecological forecasts come in. But understanding all of these pieces then allows you to think about what you're forecasting and how it feeds into decisions. Understanding how to do some of the trade-offs and how ecological forecasts fit with the decision models I think is really important. And we'll get to that on Friday. There's ways of thinking about value of information. The methods that you would use to be able to assess whether or not improving your forecast has value for a decision. And the sensitivity of the scientific information to that decision. And there's ways of then thinking about how we link adaptive management and iterative forecasting. We'll get to that on Friday. But what I want you to think about this week is think about these kinds of decisions. Think about who might use the forecast that you're generating. Think about the measures, the performance measures that they might use. Are there ways of developing your forecast and quantifying the consequences, whether they're predictions or projections, that may lend itself more easily to being able to be incorporated into a decision? It's not the case that every forecast is going to be decision relevant. There's a lot of forecasts that just improve our understanding of ecological systems. But so many forecasts could be so useful. And if we can think about how we can better incorporate this kind of approach into the ways that we're doing ecological management and really making robust choices about our environment, I think we can transform the way that we're both forecasting but how we're making decisions and improving our natural systems for future generations. I honestly believe that ecological forecasts are the frontier for how we operationalize adaptive management. So when we think about that, I'm going to bring up Holling again, because Holling wrote, modeling involves two fundamental phases. And so when we think about this, the first phase is an inductive phase. This is a creative synthesis constructive phrase. This is where we map from how we think the world works to how we are abstracting it in terms of a model. Then comes this deductive phase of like, literally, we're churning the crank. It's a mechanical phase. We're using these mathematical analyses and simulations to sort of turn the crank and develop this. So the piece that most people don't do is they don't go back because it takes so long to get that deductive engine to work. You don't go back and say, OK, let me compare what I've gotten from my models with this expectation of reality and really understand the inconsistencies and how I can learn from those processes to improve the model. These are deliberate decisions that you all are making in terms of how you improve the model. So learning is really critical in this. Decision scientists were right along with the weather forecasters. Part of the reason why the weather forecasters got better is because decision scientists were helping to understand how well those models perform because we had a natural experiment with local weather forecasters making predictions and initially, the people outperform the models. And that's probably going to be the case with the ecological forecast. If we take your predictions versus an expert group, I would hypothesize that the expert group is probably going to outperform for a while. But if we keep going at this and we think about this in terms of learning and how we improve our understanding, how we make better choices about the model, how we link it back to reality so we're being very deliberate about how we design this, we can accelerate learning from failure so that then these models are actually useful in decision processes and we can operationalize them into adaptive management processes so that as we get new data, the models are updated, but we're also revisiting our environmental management decisions so that we're understanding whether our decisions led to good outcomes and if they didn't, we're actually making better decisions because we're getting our new information, we're understanding whether it works, and we're going back and improving it. So that's my call to you all this week is to really think deeply about these things because it's not just being able to represent the system. I think it's fundamentally important for improving our decisions. So thank you all.