 So, expert elicitation is a pretty powerful process. It's a rigorous approach to be able to systematically describe relationships, whether those are probability distributions or connections between various variables that we expect, and really make our scientific knowledge transparent. It's been used in a number of cases where there simply is not data and information to be able to make assessments, but experts have knowledge that needs to be included in decisions. So, for example, some pretty famous examples of very rigorous expert elicitations being employed related to choosing between different locations for where you could store nuclear waste in the United States. And it requires thinking about tens of thousands of years from now and implications for failure of the storage containers. And when you would expect those to degrade, it involves calculating or quantifying probabilities related to how much you would expect if you were to have seepage when you would expect it to potentially hit groundwater sources or other things that would impact humans and human health. And so these are really useful techniques to employ within forecasting because in some cases we have a lot of data to be able to describe things that we care about. Sometimes we don't have as much data we can use as we would like and we can use clever statistical approaches or other mathematical approaches to fill in the gaps. And in other cases, we really need to use experts. The paper that I put in the reading list was done by Granger Morgan who has done a number of expert elicitation exercises within the U.S. primarily focused on public sector decision making. So a lot of things relevant to environmental management. And he says in his paper, done well, expert elicitation can make a valuable contribution to informed decision making done poorly. It can lead to useless or even misleading results that can lead decision makers astray alienate experts and wrongly discredit the entire approach. And so, I mean, similar to anything we do, there's probably a way of doing it right and many ways of doing it wrong. And so if you think that an approach like this is useful and that it's probably worthwhile to make sure it's designed right and potentially involve experts in helping to navigate some of those processes. Just to give you a sense of the things that need to be considered when designing this and this is not to discourage you from using this, it's more to say we use it a lot within, you know, individually within modeling processes because we are experts ourselves. But when we are going out and seeking outside expert opinion, there's just a different level of rigor that has to be applied. So you want to make sure that if you're asking a question, it's relevant to someone's expertise and that they will be able to make some kind of, in this case, probabilistic or predictive judgment. You don't want to use qualitative uncertainty language. There's a number of studies that have shown if you use words like likely or with high certainty or some of the language that's used like in the IPCC, they define the probabilities that are associated with it. But if you don't define those, what you find are terms like likely have ranges between like 30 and 70 percent. So what do you do with that when you've structured something qualitatively? There's a lot of mental shortcuts. This word called heuristics that we use a lot, that's basically, you know, these mental shortcuts that we use a lot in decision making, they can rear their heads in really unpleasant ways if you've structured your elicitation to more or less play to some of those heuristics. So you want to be aware of the ones that typically arise when working with people and keep those in mind so that you're designed to minimize or potentially mathematically correct for biases like overconfidence. I can give you a paper of ours that we did that after collecting the elicitations. Experts, you want to choose them carefully. You want to think about the protocols of how you're setting this up and how you're intuitively refining this so similar to your data, you want to have calibration checks and ways of validating it. You want to think carefully about the uncertainty, a functional form, the diversity of experts, similar to giving them what you want to say so that you're not creating a representation that's too narrow based off of the entire expert field. And you want to think carefully about combining expert judgments because a lot of times we might want to collapse probability distributions across multiple experts and there's cases where that works perfectly well. I've done it reasonable a number of times and there's cases where it doesn't make sense similar to ecosystems. Some you can aggregate and think about them in terms of more general characteristics. Some are so distinct you really need to treat them independently. So I want to give you an example because we keep talking about weather forecasting as a correlation. Decision analysts have been in the mix from the very beginning of weather forecasting. So we went from this situation where we weren't very good about it. We now have models that are able to predict the thunderstorms that are going to occur in the hail that rains down at a pretty reasonable prediction level. However, there was a natural experiment that we had with weather forecasters that we don't necessarily have with ecological forecasting. But it gives us insight into how we might be able to think about experts with ecological forecasting. So with weather forecasts from the very beginning days, even before we had models, we had local weather forecasters who were the local experts who would predict weather conditions. Now you couldn't do that very well like without some of the more sophisticated technology. But you had a situation where you had tons of these people plus you had your model predictions and allowed you to actually compare experts versus models. So starting back in 1965 when a lot of this started, you could look at an expert, a weather forecaster versus a weather model and see which performed better using squaring rules. It worked pretty well for a while. And that's because a lot of the models weren't able to predict as much as what the experts knew and we're not necessarily able to do that at the local level. You had a lot of local knowledge that was being incorporated into those probabilistic assessments that couldn't be captured by the model. Over time, over two decades or so, the model's caught up. And actually I think about in the mid-80s is when the model started outperforming regularly most weather forecasters. And so I think this gives some insights into how we think about it in terms of forecasting because we're still using expert knowledge. So this is one that pulls from multiple different data sources but how they think about aggregating is not necessarily using models even though you have individual sources based off of models and data sources. They actually have an expert process that develops these qualitative scales related to whether or not drought is persistent or drought improvements are likely and things in between. Now I would argue that there could be a greater transparency with the process that they used to make those judgments but it is an example of how we have experts operationalize into forecasts that are used for a lot of decision making processes. Which means I would argue especially at this stage where ecological forecasts are really nascent that if we think about how we combine ecological forecasts with formalized expert judgment really understanding the experts assessment of a particular problem that we're trying to understand and solve that we can actually get better ecological forecast and that that might be something that is useful to operationalize as we work to improve the skill of the models over time. So we don't have to wait for models that perform slightly better than 50, 50, 50. We can get predictions and forecasts on time scales that are decision relevant by thinking about how we combine these more effectively. So the three major ways that I see expert elicitation being particularly useful within forecasting is one is we don't need to use uninformative priors. There's a lot of cases where we have existing knowledge and we can construct an informative prior and expert judgment even your own expert judgment going into those kind of modeling context in a formalized way can be a really powerful way of including it within these kind of Bayesian forecasting approaches. Likelihoods. So this is not dissimilar from the weather forecasting situation where they use the model as a prior and they had local information you got a posterior distribution that was combining those. In other cases a lot of times in the models that I've constructed we have questions whether there simply exists no data or no data on the phenomenon that we care about. And that means that expert elicitation can be a really useful way of sort of bridging the gap in constructing the likelihood when you have an uninformative prior and you really need data. The other is so these are constructing probability distributions and we're going to do that in a second. The other thing is is we've talked about you know these relationships between various variables in the causal structure that we're thinking about and there are approaches a conceptual model influence diagrams mental modeling where what you're doing is you're capturing your expert knowledge structural equation modeling you're capturing your expert knowledge and formally representing it in a causal structure of that that model and formalizing it and formalizing the choices that you're making because there's trade-offs with any of these modeling choices we're not able to represent absolutely everything and there's a reason why models penalize complexity. Those are those are some entry points but what we wanted to do was actually construct a probability distribution. So I'm going to invite Mike up here to talk about constructing a prior related to the travel time from Boston University to Logan Airport. Yeah I've done this example in both my graduate classes with the forecasting class and my Bayesian class and I've run into students years later they remember nothing about Bayes Theorem but never miss flights anymore and I know not all of you are most of you have not lived in Boston so I'm going to rely on some of those folks in the back who do. I'm also feeling a little nervous doing this in front of Melissa who is a bona fide decision scientist because she started she started with the quote about how done poorly. I was like I've never gotten formal training in this I've just been reading up on it I'm like she's going to just tell me I've been doing it wrong for years. My understanding which is all started you know as book knowledge really reiterates the point Melissa made about some of the cognitive biases and the importance of doing elicitation in a way that tries to avoid those cognitive biases and we the place that I have used elicitation most often has been in the construction of priors I feel like if you read the literature when you're trying to learn Bayes there are an abundance of uninformed priors used all over the place and if you think about that well statisticians don't actually know how anything works in the world they don't know how your system works they don't have any expert judgment so of course they're going to use in for uninformed priors but you do you know a lot about how your systems work and I've but I've seen so met so much of the ecological Bayes literature is just ripe with misapplication of very very broad priors on things that we do know a lot more about on the flip side I've seen particularly in the process based modeling literature a lot of examples of one of the most important cognitive biases with elicitation which is anchoring so if you have a process based model that model has default parameters if you ask someone who's been using that model for a few years or developed it for decades to construct a prior they will center their prior on the default parameters they will then set fairly narrow bounds around that and often they do it uncritically I've seen tons of papers where people's priors were the default parameters plus or minus 20% which is yeah you're you're way overcome for some of those parameters if you actually sit back and say what's the biological range of variability that this process is it's not 20% you don't actually knew that but I've been through this exercise with with you know folks that you know build these global system models and saying okay let's step back and think about what what values parameters can take on and one of the the ways that I've done this has always focused on building a probability distribution but building a probability distribution in terms of the CDF the cumulative distribution function for example if we have a normal distribution you know centered on zero standard normal variance one that's our you know PDF but the corresponding CDF for that kind of looks like a logit and it's centered on zero you know minus one to one you know it's the integral of our PDF and for various reasons it's often easier to work through the logic of how the CDF might be structured and then having constructed that CDF to essentially take the derivative of it to get back to the PDF so if anchoring is one of the key cognitive biases anchoring starts by thinking about the middle I think what we often want to do when thinking about constructing priors is to instead think about the extremes so we're going to do in constructing a CDF is going to work from the outside in so in a few days a lot of you will be leaving Boston and need to get to the airport you're starting here on campus or fairly close to campus what's the minimum time that it can physically take to get from here to Logan under the absolutely positively best possible case disregarding all state and federal law and no traffic and any form of transportation yeah no limitations yeah well we know for one thing it's positive we're not assuming that we can have a time portal and go back and we can get to the airport before we left though there's many days where I wish I did but yeah it's you know okay so for reference if you don't drive on the roads it's a little under five miles to campus from campus to the airport so maybe you you know you hop on you know your private helicopter it's a you know Apache Tech it's going cruising it's going to take yeah a minute or two so let's say the lower bound might be around one minute is there an upper bound on how long it takes to get to the airport given that the Boston traffic can sometimes suck and there is a background mortality rate you might argue there is not there is that you might not ever make it to Logan the maximum amount of time like yeah you know walking there like how long would it if you're walking five miles is that you know out half but it could be raining you could be snowing I've I've tried to walk down the street out there when there was literally 10 feet of snow so assuming that you did not die on route you know it still could take say tens of hours at an extreme you know that's is your worst worst day nightmare but we say that worst nightmare doesn't actually happen very often it's it's the asymptote of a probability distribution it is out in the tails and it's really hard to get out of that like this is a really scary situation how to get to like a realistic model or understanding of what the system is that bias has a name availability bias yeah availability bias so it also depends on your framing because you know if what you're trying to do is like what I'd say blue skies planning so you're trying to preemptively think like we are trying to manage for various species and we're thinking about introducing feral pigs with this idea and you talk with different experts I would approach it two ways and this is usually how I construct my elicitations the first thing I do when I meet with an expert is actually I sit down and talk with them about that system and I get them to just literally explain it to me I construct a conceptual model how many of you think in box and arrow diagrams okay yeah so there's a couple of you and if you think in box and arrow diagrams like you have a really powerful skill set because when people are talking you're literally like and drawing these that's that's what you're doing during that kind of phase is that you're literally having someone explain to me you know the relationships for a phenomenon that we care about and you're sketching it out and then you're stopping and just checking this is what I'm hearing is this the way that you conceptualize it you know which of these are some of the dominant drivers which are the minor are there other factors that you would expect that would influence it the whole idea behind that is to do exactly what you were talking about Jen let's get people like sort of outside thinking thinking both about the dominant things that are likely to affect that system what they would typically model but also about these other factors that would influence it you know both environmentally but also the the human social factors that may not be included like in an ecological forecast but absolutely you know in some cases affected when we're doing projections and we need said you know boundary conditions then after we've done that and I've constructed a conceptual model they've had a time to reflect on it I've maybe even sent them a summary so if they have other ideas they've been able to do it usually in a separate session that's when I do a probabilistic assessment because you know when when you do that and ideally with the conceptual modeling and with other things what you want them to do is also connect the dots so we're thinking about you know the effect of the brown snake on the crow on everything but the crow in particular and I was I was trying to think of the Marianna Crow you know what are some of the other invasive species examples you know what do we think might have similarities or differences what are the invasive species introductions that have worked out well which ones have which ones have worked really poorly what have been some of the unanticipated effects so that then if you're constructing you know some type of probability elicitation over something you care about you're getting them to think about it you know both in terms of those those extremes and what what is most likely to happen that is easier to do and blue skies planning when something hasn't happened once something happens you know something oftentimes it's bad that constraints things a bit they're no longer trying to forecast if the event is going to happen well they're not trying to forecast if it's going to happen they're trying to forecast the effect the effect of what will happen and that's that's a slightly different situation and if you're doing these intertip forecast in some cases it can be there are not many examples of long-term systematic expert focus forecasting the drought monitor outlooks are one example where that uses experts regularly like on a weekly basis to think about that it's really expensive and time-consuming and large federal agencies maybe fish and wildlife service are the ones that really have the capacity to like operationalize those kinds of approaches interest of time I'm gonna have to nip that in a bud because I thought we were gonna spend half an hour on this we spent closer to an hour on this but I'm gonna get you guys started on might instead say well realistically let's think about say 99% of the time so we'll get rid of that last 1% of truly abysmal cases and think about well 99% of the time what are the odds well 90% of the time 99% of time your car is not going to break down you're gonna get stuck walking but there's gonna be some pretty bad weather days and pretty awful rush hour traffic that you could be stuck in where you're crawling along so let's say 60 minutes and so we would say at this 99% for a 99 percentile I mean I would I would think maybe more like two hour and a half two hours if you ran into say you had an accident that shut down all lanes of traffic on and driving towards Logan two hours it was slow yeah it was an hour and a half for me yeah I guess yeah like getting on hopping on the tee and then it like breaks down I've I've tried to get across Boston where you walk out here in a socks game just let out or you know you're trying to get out and yeah yeah it could be bad so so one of the things you might do during elicitation is say what's this what are these extremes but then you might then ask yourself okay what could cause something to be worse than that extreme so it's useful to have checks to say okay that thing that we wrote down what could cause us to be worse that you know to be to be skeptical of where you start to kind of revisit the point you've put down and think what you know what am I missing you know I'm not just talking about a bad day I'm talking about a really bad day so the process that we would go through would be to march inward so we might go from 99 to say 95 percent and say you know one in 20 that's a pretty everyday bad day that happens what's well so one of the things that I found useful in doing these kinds of things is when you work with experts especially with like these this kind of example of the airport is you saw me go like straight to an hour and a half because that's the worst experience I personally have had and usually a lot of times experts think of their I mean they're experts so they're thinking of like what they're familiar with and so you might have something like you know an hour and a half and Mike's absolutely right you basically then push them to talk about the extremes tens of hours probably like your your your true extreme bound if you if you were creating creating bracket so maybe not like a realistic bound but it but a true true extreme extreme bound because I think most people would switch modes of transportation before it got to that but when when I do this usually what you do is you might start here but instead of stepping them through this I actually do this yeah I was going to switch to the other extreme after that yeah switch to the the other one and the reason is is that then you force like them to think about it in in a different way so you're flipping flipping it so that then they don't end up focusing too much on one part of the distribution without thinking about you know what's the best case scenario and that then when you ideally get to the middle it really does match what their like expectation would be so 95 I mean I can easily see that being somewhere in that hour and a half to hour and 15 minute range so maybe that maybe we'll say that's you know maybe 80 minutes and then like Melissa said we might now switch here because here this one's sitting at 0% and we might say you know choose say 5% fastest time we don't actually have our Apache attack helicopter anymore we're on the road we're driving but it's it's two in the morning there's no traffic and you are not regarding laws particularly well you know 15 minutes yeah even you can you could make it in 15 minutes easy you could probably make it in a little bit faster yeah so maybe that takes me let's say maybe 12 minutes and I start connecting the dots so might do there I might jump up you know in the interest of doing this getting this done I might jump up to say 25% that's a pretty that's my you know now thinking about intercourtile range that's that's a pretty typical day I'm gonna hit some traffic so I might then say well along the road if I'm driving it's more like six six and a half miles on a typical day I might be doing on typical but better than average day I might be averaging about 40 miles an hour and what does that work out to 20 minutes 20 something like that so maybe that's a maybe that's a 20 minute day and then I might come back to the other stream and say well what's the other side of that intercourtile range it's worse than average it's it's you know it's a typical Boston rush hour you know I'm going 20 miles an hour so four minutes per mile five miles uh six miles seven miles whatever uh not doing good math on top of my head it's late in the day yeah maybe 40 minutes and then I guess the one of the really important things here is we're we're estimating that central tendency last we're not estimating it first so so if anchoring is a problem that idea of starting with the what you think is the most normal thing and then moving away from that works we've done the opposite we've come from the extremes back in so the last thing we estimate is a mean you know so maybe uh maybe that's 30 minutes and so now we've got a set of data points that describe a cumulative distribution function that now corresponds to our eight you know before collecting data just using our personal experience what we think the probability distribution describing getting to the airport would look like like I said we convert this convert this to a pdf and if I'm doing this if I'm using this for some sort of formal Bayesian inference one of the things I will then realize is that all of our Bayesian tools like name distributions so what I might then do is is take the start thinking about well what's the candidate set of probability distributions that conform to being capable of capturing these shapes well I would then say I definitely want to deal with zero bound distributions I'm not going to fit a Gaussian to this I'm going to want to deal with something like a gamma or a log normal or y-ball something that's a zero bound distribution I might have a set of four or five six probability distributions that have this nature of being zero bound you know even if I haven't done this formally but I can tell you if I sketch this out you're going to end up with a skewed distribution that's zero bound you can just see it because this tail was much longer than that tail and so I have data I have points here and I might fit all five or six possible distributions to those points and and see which does best and I might then end up you know translating this into something like you know a a gamma with specific parameters that then I can use as a prior when I'm constructing my model and you know just to just sort of think about this so this is sort of basically Mike's expert PDF for the amount of time he thinks it would take to get to to Logan if I were looking at my central tendency given the embarrassing amount of times that I've traveled to Boston University and not see Mike you know my average travel time is about 15 minutes because I usually take the tea in because I'm coming during rush hour and so part of this you know is expert dependent what you will likely get is for a question like this the distributional form is likely to be the same but you might have slightly different central tendencies or how that distributional form is constructed but if you're having similarity that's where to potentially a good case of doing an expert aggregation so if we ask you know five different people to and constructed their PDFs for this and then aggregated or more likely to get a distribution that is the true broader range and something that is probably more likely to sort of the central tendency of that that average condition come back to a couple of my experiences and working with modelers is I found that things like this are remarkably common when I go through the process of talking to them about their parameters you know there's lots of ecological models where the parameters we deal with and the processes we deal with are zero bound and skewed and you know when you then look and say well why is the whole literature we're using using symmetric Gaussian priors uh and then also that the variances are are wide like if you went to the literature right now and said you know what was the statistician have done you know the statistician would have done like normal mean zero variance of ten to the sixth you're like really there's a 50 probability that it's negative one million minutes to get to the airport like this is the absurdity of some of the default priors we use when when you actually start thinking about a problem you know you realize that you really do have a lot more constraint on problems than the default priors that you slap on if you're not stopping to think that said you know data will always given enough data you will always overwhelm your priors but it can be really handy to spend just a even if it's just five minutes kind of internally eliciting what you think about a problem can often give you much more reasonable constraints on what you're what you would what parameter values you would you believe if you got back so that if you run a statistical model and then it comes back you get an estimate of a slope or an intercept and you go that's impossible and the question would be then why did you assign it probably prior probability if you a priori didn't believe that result was possible why did you give it non-zero probability and that can help constrain models and and if we come back to thinking about forecasting and the types of forecasting problems we're going to face you know I one of the things I said yesterday is that I do expect there are going to be certain types of forecasting problems that will be chronically data limited things like emerging infectious disease invasive species problems where there's no way we could have enough information yet and those are ones where elicitation is going to be really essential to producing that first-order initial forecast that we can then update as data does become available yeah so when you think about this there's sort of the two things that we we've talked about so one is how can I as an expert in my own right or with my my team of modlers really think about this you know in a rigorous way both to add information a priori that helps us to build better models but also so that when you get output you understand when it might be too narrowly constrained because the way that the model is constructed is only looking at a piece of the larger system that you care about so that then as you think about it and you think about that output you can contextualize what those results mean because if you guys aren't doing it there's people that are going to be using those and potentially using them incorrectly because even if you're doing like a full uncertainty characterization it's looking at a very narrow piece of the larger system that you know that's being informed for a decision you have to contextualize those model results and this kind of process can help you to have some of the language to to put that into into context and in some cases and I you know when when you are in data limited situations bringing in someone who has rigorously done expert elicitation so you can have some of the data to rigorously parameterize your model and and predict some of those phenomenon has been proven time and time again as an effective way of being able to sort of fill in the gaps until we can collect the data and in some cases we will never be able to collect the data because it's unethical so in some cases there there's literally no way we will be able to do some of the things that we want to be able to predict unless we use elicitation and to pick up my last thoughts on this idea of contextualization having pushed process modelers through elicitation on their own models I found on more than one occasion the discovery of bugs in the model I've had seen them have to go look up the units on the parameters I've seen them have to dive into the code and make sure they understood how the parameter was used and then come to the realization that there's not the way they wrote down things didn't actually make sense to them once they stopped to try to put constraints on the problem and they have also seen that that process of going through an eliciting eliciting what they believe to be plausible parameters a prairie really made them stop and think about the process so you know even if the prior gets overwhelmed you've stopped and understood the process better you understand your model better this isn't always terribly critical with a simple linear model but anytime you're dealing with a nonlinear model a process based model mechanistic model stopping to do that elicitation actually will give you will make you stop and really understand the model you're working with which is is actually been a really valuable exercise