 There was a nice tweet from Ethan Whiteslab last week, I think, where they got a review back saying, this is all great, but why should we forecast as ecologists? Why bother? So I wanted to give a perspective on why we should bother and why, particularly I think iterative forecasting is an important direction for ecology to go. So as Melissa said, decisions are fundamentally about the future. So one of the main but not by no means only reason that we were interested in ecological forecasting is because there are clearly pressing societal needs to make better environmental decision making that range from high-level national international policy down to everyday decisions made by individuals in the public. All decisions are fundamentally about the future, and if we want environmental and ecological understanding and knowledge to inform those decisions, one way that's really important is for those to inform predictions and projections, because ultimately you don't make a decision about what happened in the past, you make a decision based on what you think is going to happen in the future, and a forecast is making explicit what we think is going to happen in the future. So part of the reason we make forecasts is to make our science more relevant and to respond to the clear societal need to make better environmental decision making. That said, I've come more and more around to the opinion that ecological forecasting is a win-win not just for better decision making, but for better basic science. So one of the emphasis of Nefi, the name, near-term ecological forecasting initiative is on the near term. As I said in my intro, I've spent most of the last decade of my career focused on models that are making projections out to 2100. I feel like the ecological community, when they think about forecasting, has gotten fairly good, not necessarily good at making predictions, but fairly good about thinking about projections out on the timescale of 2100. But after spending a decade making projections out to 2100, I don't know anything more about whether I'm any good at making predictions out to 2100 because it's not 2100 yet, and I have not validated whether I'm actually any good at that. By contrast, I've come around to the perspective that if I make a projection for next week, I know pretty well, pretty quick, whether I'm any good at making a prediction for next week. And for this season, the inter-annual variability of the next few years, the phenological variability over different seasons, these timescales are actually timescales where we can actually validate whether we're any good at making predictions. And I've also come around to the perspective that a viewing forecasting, an iterative forecasting in particular, fundamentally is an expression of one way of thinking about how the scientific method itself works. So in the scientific method, we start by, you know, we form a hypothesis, we use that hypothesis to make a prediction about what we think is going to happen in the world, we test that hypothesis against what we actually see in the world, and we interpret the results. Well, in the iterative forecast, the models we use to make forecasts are, at their essence, a distillation of our hypotheses about how we think a system works. And for me, I think that's an important perspective to have when you think about models. Models are essentially just the formalization of our hypotheses. So if a model is our formalization of our hypotheses, then we use those to make predictions. Those are an expression of our hypotheses about how we think the world works. I think in ecology, we have let ourselves off easy a lot when it comes to formulating hypotheses and making predictions. I feel the majority of the theory we work with is very qualitative. I feel like the majority of the predictions we make from those theory are fairly qualitative. And the way we test those are often not super strong tests. By contrast, using a model to make predictions actually puts a quantitative number on what you think is going to happen. And with acknowledgments, you will probably be wrong most of the time. But when you're wrong, you learn something. So we test those hypotheses by making predictions. We interpret this. And one of the things I've been thinking more and more about is by making iterative forecasts, by continually making predictions, we continually get feedback about how we're doing. And so the idea here is the belief here is that by making near-term forecasts on an iterative basis, we have the potential to actually accelerate the pace of science. So not only make that science more relevant to society, but also try to make the pace at which we improve our understanding accelerate. And for me, I showed the Freedling-Stein diagram earlier, one of the things that was striking about that was that if you look at the 2007 version of that and 2014 version of that, the biggest difference in seven years was the color scheme. They were otherwise pretty much indistinguishable, suggesting that in seven years of effort, the global trestle carbon cycle community essentially didn't get any better at what they were doing. That at a time with rapid environmental change, we cannot rest on our laurels. We cannot just wait to get better. We need a way of getting better faster and responding to these needs faster. So one of the things that I think, one of the ways that I think forecasts help us do better basic science is because forecasts are a priority. So we are actually saying before we see what happens, if we make a prediction in space or a prediction in time, that we don't know what that data is at the time that we're making the prediction. I think this is in contrast to something we've seen in a lot of sciences, not just it's been very prominent in fields like psychology, but I don't think ecologists are in any way immune to this, is that there's been problems with reproducibility of results. There's been problems with accusations of p-hacking and accusations of overfitting of models. And I think with a small percentage of times those are deliberate. I think in the vast majority of times those are completely inadvertent. But when you have all of the data that you have in hand, it's very easy to just keep fitting models and testing models and you end up testing all these hypotheses and you can end up with a model that fits data great, but you have no idea if it actually is reproducible, if it's actually general. By making your predictions a priority, I think it is in some ways a stronger test because you don't know. It's much harder to overfit when you don't know what's going to happen at the time that you're making the prediction. It's not perfect, but I do think it provides a stronger test. Also think that trying to make forecasts will force us to synthesize our understanding of ecological systems better. So to give an example, like I said, I started as a plant ecologist. Plant ecologists and their brethren in agronomy have been making nitrogen addition experiments for over a century now. We dump nitrogen on plants. Hypothesis. Nitrogen affects plants. Stronger hypothesis. Plants like nitrogen, they grow faster. Great. Does anyone not believe that hypothesis? It's pretty much come up consistently for over a hundred years that when you add nitrogen to plants, they tend to grow faster. If you do a nitrogen experiment today, you start it today and you use traditional statistical methods, your null hypothesis still remains nitrogen has no effect on plants. When you get a p-value, it's refuting the hypothesis nitrogen has no effect on plants. You did not believe that hypothesis before you started. So why in the world would you use that as the basis of your model? By contrast, let's say we actually went back and synthesized that 100 years of research on nitrogen addition experiments and came to the conclusion that in this particular experiment when I'm dumping a specific amount of nitrogen on a specific crop in a specific location and I say, based on what I understand, I predict 25% increase in yield plus or minus 5%. I've now made a quantitative prediction. My hypothesis with a number around it and an uncertainty around it. Under the null hypothesis, a 25% increase in growth and a 50% increase in growth are indistinguishable. They both refute the null hypothesis that nitrogen doesn't affect plants. By contrast, 25% supports our existing understanding of the system, 50% rejects our computing. You know, it says something is missing in our computing understanding. Most statistics don't just give us dumb null hypothesis. They don't let us realize when and we have and have not learned something new. By contrast, no response of nitrogen is nonsignificant under traditional hypothesis testing. But if this is what you expected, it is a very novel result. If I dumped nitrogen on a system that my current understanding says is nitrogen limited and it doesn't respond, that to me should be a highly significant result, not a nonsignificant result. Yeah, to reinforce so, at the moment, we're not very good at this. And one of the reasons, like I said, we're not very good at this is because when we make predictions over a long time scale, we're not getting that feedback that we need to learn. So if you look at any field that's gotten good at forecasting, whether it's weather forecasting, economic forecasting, we will set aside whether they're actually good at it or not. But any sort of field that whether it's in the natural sciences, biological sciences, social sciences, any discipline that gets good at forecasting, it's very clear that that requires feedback. You need some learning process in places where we make predictions and we don't have a process of getting feedback about how we're doing can result in us being very overconfident, falsely overconfident in our ability to make predictions because we do it a lot but we didn't ever get any feedback about whether we're good at it. And as was highlighted in Colin's video, the experience has been in fields like weather forecasting is that that process of getting feedback does result in improvement. I mean, there's a lot of things you could lay at the feet for why weather forecasts have gotten better. I mean, they do have more powerful computers, they have more observations. But I genuinely believe a lot of why weather forecasts have gotten steadily better over decades is that every day, well technically six times a day at this point, they put out a global hypothesis about how they think the world works and then they confront that hypothesis with reality six times a day over the whole globe. If we were doing that, we sure as heck would learn a lot about what we do and don't understand. So to me, the question that I've been thinking about is can we forecast ecology like we forecast weather? So what I want to spend the rest of this chunk of time before morning break talking about is thinking about this from a theoretical perspective, give a quick introduction to some of the methods we'll learn about this week and most importantly emphasize the idea, the importance of thinking probabilistically when it comes to trying to think about how we bring models and data together and how we make forecasts. So if you, this comes from the book, but if you had never worked with models or tried to make forecasts before, your perspective on modeling probably consists of, well there's data and there's theory, we put data and theory together, we make a model, we run that model in the forward, we make a forecast and then that miraculously somehow informs decisions. Life is never that simple and we'll go over the course of the week, many of the components of why life is not that simple, but one of the ones that I think I really want to emphasize is because everything that goes into these models has uncertainty associated with it. There's uncertainty in our theory that reflects into uncertainty about how models are actually structured. Unlike the physical sciences, we don't have physical laws behind a lot of our models, a lot of them are empirically inferred, there's therefore uncertainty about the parameters, there's uncertainties about the drivers, uncertainties about the current status of the world and uncertainties about any of the scenarios that go into making projections. Because of that, those uncertainties compound and we result with uncertainties in our projections. If we think about trying to make forecasts, one of the questions will obviously come up is are we any good at this? How do we measure the predictability of different ecological systems? And one of the things I'm going to put out as one of the important measures of predictability is the uncertainty in those predictions. Rule of thumb, when you make a forecast, the uncertainty should increase as you go into the future. If you become more confident about the future than you are at the present, you've probably done something wrong. So we're going to assume, and there's some math behind this, uncertainty is going to increase as we go into the future. So one way of measuring predictability is not just the uncertainty in the prediction but also the rate at which that uncertainty increases. A process that becomes where the uncertainty increases rapidly is one that we're going to have a shorter horizon that we can make effective forecasts over. A process where that uncertainty grows slowly gives us more time over which those forecasts are likely going to be effective. So how do we characterize systems as being predictable or not? So throughout this course, I'm going to emphasize the use of dynamic models in forecasting. I understand very clearly that they are not the only class of models that we use to make predictions but I think they are particularly common in pretty much every sub-discipline of ecology where we often think about predicting the state of some state of the system, some y, at some point in the future, t plus one, that's some function of the state of the system right now plus some covariates or drivers given some parameters and some uncertainty. So if I take this basic framework, I can break down the uncertainties into the future, but what you can do is you can break down each of those things that go into that equation dynamic model in terms of their contribution to the overruns on certainties. We end up with a contribution coming from the internal dynamics of the system, these external drivers, and then what I'm going to often do is separate the uncertainties and the parameters in a model into two parts. What I'll call the uncertainty about the parameters themselves often, what is the mean of this parameter and distinguish that from the heterogeneity and variability in processes that cause heterogeneity and variability in parameters, often what statisticians would call the random effects. So being able to separate what is the mean of this parameter from how does the process and the parameters vary in space and time, and then just the remaining unexplained error, and I'm also going to often separate out a traditional residual error into observation error as separate from process error. So observation error is the error in the data themselves, which is separate from errors in the process, the fact that our understanding of systems is always incomplete, so there's always something our models do not capture, and so that's what I'm going to mean by process error. So each of these five terms follows a very similar pattern whereby the uncertainty, the contribution of that term to the future involves two parts. The uncertainty about that part, so say for example the uncertainty in the initial conditions translates into the uncertainty in the future, and then the sensitivity of the projection to that particular item. So every, that particular term, so every term involves this, what is the uncertainty about that component and what is the sensitivity to that part? So yeah, we have, you know, the internal component involves uncertainty about the state, which is the initial condition uncertainty, and the sensitivity of the forecast to that state, which is actually equivalent to what we all learned in intro ecology of system stability. To make this a little more concrete, I'm also going to, so what I'm going to do next is I'm going to kind of walk through the implications of each of those uncertainties for how we make ecological forecasts, but in doing so I'm going to rely on a very common model that we've all, again all seen since intro ecology, the logistic growth model, as an example for how we characterize uncertainties and propagate them. I'm also going to kind of, one of the things we're going to see in the course of this, over the duration of this course, is kind of the difference between fitting models as processes, fitting them as dynamic time series processes with uncertainties, versus what you would get if you fit that model as a function. So if you take the logistic growth model and you just treat it as a function and you fit it to this data in black, you know, the green line actually ends up being the best fit, which is kind of amazingly bad. By contrast, if you fit that model as a process by which populations grow iteratively over each year with uncertainties, you know, you can actually, you know, your best fit, you know, you can make it to follow the data, but again that's because you have access to the data. I order each of these terms and how they affect the predictability of ecological systems. So like I said, the first term has to do with internal stability of the system, where this idea of stability is the classic one we've all seen thousands of times as ecologists. Ecologists, as an ecologist, I, like anyone else, cannot resist the urge to draw pictures of balls rolling into valleys and off hills. So one take home, everything we learned in like introecology theory is actually relevant to forecasting, but it's not the only thing relevant to forecasting. But it does matter because the stability of systems does affect their predictability a good bit, as well as combine that with our needing to understand what's the state of the system right now. Qualitatively, we can divide that distinction between stable and unstable into making, you know, kind of two different predictions about how our forecast uncertainty is going to change over time. So this is time on the x-axis and this is how the predictive uncertainty is growing. If the system is unstable or chaotic, that predictive error is going to grow exponentially as we move forward in time. This is exactly what characterizes weather forecasts. The atmospheric system is unstable, it's chaotic, and therefore the uncertainty about the current state of this atmosphere will grow in time and come to dominate the forecast error. It's actually because of this realization that the atmosphere is chaotic that one can characterize atmospheric weather forecasting as an initial condition problem. So the nature of the forecasting problem in atmospheric sciences is that initial condition problem. If I want to reduce uncertainties in my weather forecast, essentially the way that I do that as I reduce uncertainties about the initial conditions, because of those five terms, that is the one that dominates weather forecasts. I posit that is not the case for most ecological forecasts. So one really important distinction between ecological forecasts and weather forecasts is I think the relative importance of this term relative to the others is very different. That said, I do think there are places such as emerging epidemics in ecology where there is evidence that systems are chaotic, where this high sensitivity of the initial conditions really does matter. So there may be cases in ecology where that initial condition uncertainty dominates, but I do think most of ecology falls in this case of having some form of stabilizing feedbacks. I might not know exactly the state of the system right now, but I know as it evolves through time that there are stabilizing feedbacks that keep those bound. So if I think about processes like succession, succession has stabilizing feedbacks whereby trajectories of systems, their uncertainties are bound. They don't diverge exponentially. Other thing to note is this internal stability term, because it is a feedback, is the only one that grows or declines exponentially. To a first approximation, all of our other uncertainties grow linearly. We don't know their relative importance. So here's an example in the logistic model. You know, if we have uncertainty about the initial condition, this is kind of a constant envelope showing the overall uncertainty converging as you move to that carrying capacity, and these dash lines are just individual ensemble member realizations of that process. One of the things that's interesting about the logistic model is it also has the capacity to be set parametrically into that chaotic range as well. If you take this model and put into chaotic range, then this is just one realization. If I make an ensemble projection with some amount of uncertainty in the initial conditions, we can see it almost looks like you're doing pretty good and then suddenly you have no idea what you're doing. But in practice, I'm starting from a very small uncertainty, and that uncertainty is growing exponentially. And because I'm making a prediction, the mean of that essentially just goes back to the background means. So kind of at this point, you're not doing any better than the ecological equivalent to climatology. I know what season it is. I know the range of variability this process can predict, but I don't know anything about where in that range I might be, and that's decaying at an exponential pace. And if I want to predict this process out further in the future, the way I do that is I need to crank down that uncertainty about the initial conditions. And if I crank that down, I can make predictions further out, but again, they eventually reach some limit of predictability. So an idea of what's the limit of predictability is actually a very much more easy to define in systems like weather forecasting because they so rapidly go from, oh, it looks like I'm doing pretty good. So I have no idea what's going on. In fact, one of the classic proofs by Lorenz is that the weather itself has a limit of predictability of about two weeks. You can crank down those uncertainties really tight, and they will still blow up after about two weeks, which is in some sense the distinction between weather forecasting and, yeah, so like I was saying, one of the things that we saw is that weather forecasting is this initial conditions problem. It blows up exponentially. Because of that, everything that weather forecasters have done over the 60 years, almost 60 years that numerical weather forecasting has existed, once they figured out which uncertainty dominated, they've essentially invested tens, if not hundreds of billions of dollars on constraining initial conditions. Everything weather forecasters do how they build their models, how they deploy networks of sensors, every satellite they've launched into the space. Yeah, some of them make pretty maps that guys on TV point to, but that's not why they have those satellites. Those satellites exist for the sole purpose of feeding data into an initial condition constraint. That's what they do. That's how they do forecast. Knowing what dominated the uncertainty drove this whole field into the direction of knowing this is the thing as a field we need to do to improve our predictions. That's part of why I think there's an analogy in ecology. We're trying to make forecasts and we don't know what type of forecasting problem we have. Second term was the exogenous stability, so how sensitive systems are to their external drivers. One easy take home message from this is that those systems are more predictable if they are insensitive to environmental drivers, or there's not much uncertainty in those environmental drivers. When does the latter happen? I think the latter happens in systems where the environmental forcing is predictable, such as diurnal cycles, tidal cycles, annual seasonal cycles. Systems that are really locked to strong cycles are going to be ones that are more predictable because there's not much uncertainty. I can run solar geometry calculations out over billions of years. I know where the sun's going to be until it explodes. It's not going anywhere. The tides are going to be predictable, seasonal cycles, annual cycles, things like that. Another important take home from this, and I'll classify this one as under, even if you don't ever make ecological forecasts yourselves, it's important to understand how the data we collect informs ecological forecasts. What does this term tell us? Well, it tells us that to understand the uncertainty in ecological forecasts, we need to know the uncertainty in those drivers, in our inputs, and the sensitivity of that system. Well, what is a sensitivity? A sensitivity is just a slope, right? Well, if to make forecasts, we need to know what those slopes are, we need to collect data that allows us to quantify what those slopes are. In many ways, if you want to make a forecast, the question is not does x affect y, but how much does x affect y? So if you're doing an experiment in an ANOVA design where there's controlled treatment, allows you to answer the question does x affect y, but doesn't allow you to quantify how much. It doesn't allow you to estimate that slope. So one of the things that I advocate, a very simple thing we can do to advance ecological forecasting is to make more use of regression designs and experiments because they allow us to get this quantification of what is that slope? What is that sensitivity? The other thing we need to remember is we need to know this uncertainty about the drivers, which comes back to you need to report the uncertainties in your data. This is something I will again reinforce throughout everything we do that to be able to make forecasts, we need to know the uncertainty in the data and we need to know the uncertainty in the model. Without the two of those, we can't make forecasts. And as we've gotten better as a field at making data more open, we very often have not gotten good as a field as figuring out how to report the uncertainties about those data. This is actually one of the areas where I think Neon is really pushing us as a field is figuring out ways to report, to quantify and report those uncertainties as part of the standard practice. Other thing we learn, if we have a covariate in our prediction, we need to be able to predict that covariate in the future too. So whatever drivers are, we also may be able to forecast those. And so, you know, for example, in ecological forecasts, a lot of the things we care about do depend on weather and climate and so are tied to the predictability of those systems. It does point to a few interesting possibilities though. For example, you might actually use a different set of covariates to forecast a system than you might use it to post hoc explain the system, because you might have a variable that explains the system very well, but that thing itself is unpredictable. You know, economic past economic activity might explain logging, but good luck forecasting future economic activity. I think that's even harder than forecasting climate. So maybe you rely on other proxies that are more predictable. Other important take home, we all in stats probably learned something about model selection, you know, probably heard AIC, this, you know, whatever tests, you know, there's tests that choose between different models of different complexities. And in all forms of model selection, explicitly or implicitly, there's some penalty for complexity. This term, the uncertainties in the covariates is in none of those standard statistical model selection terms, which means if you're trying to make a forecast, and there's uncertainty about your covariates, you are not including that uncertainty, and therefore you are always selecting for models that are too complex. Other important thing is because you have to forecast X in the future and because uncertainty about anything increases as you forecast in the future, the relative importance of this is expected to increase. Because you're tying, you're tying your uncertainties to something else whose uncertainty has to increase with time, because that was one of the things we started with. The other thing I think is useful to think about in these first two terms, this internal dynamics versus external sensitivity is that it really in many ways brings us back to again, classical ecological theory and thinking about endogenous versus exogenous forcings as not a dichotomy. But as a continuum, what is the relative importance of these two factors in the predictability of a system? And I don't expect any system to be 100% internally driven or 100% externally driven, but you have two terms that tell you how the relative importance of those two factors. Okay, parameter uncertainty. I think this one is relatively straightforward, because it comes back to basic stats 101 concepts. As you increase your sample size uncertainty about parameters declines under traditional sampling theory, which doesn't always apply to every model. But under traditional sampling theory, you end up with a one over square root of n, square root of n in the denominator. So this error comes down, it comes down predictably. So for a lot of problems, as long as there's sufficient data, the parameter should uncertainty should not be the dominant problem. But there are always going to be certain classes of ecological forecast problems, such as emerging infectious disease, invasive species, where they will always be data limited problems, and therefore will always be parameter limited, will always have parameter error as one of the things that needs to be taken into account. So if I deal with carbon cycle models, you know, I've got, you know, the Lancet archive has petabytes of data. I can constrain parameters down to negligible uncertainty. But if I'm trying to forecast, you know, some emerging infectious disease or invasive species has never been here before, I have very little data on how that system, how that system is going to behave. So just again showing with a logistic case, you know, if we have uncertainty about our parameters, we can sample over that and generate uncertainties about how those uncertainties propagate. And this is, should be the part that we're most familiar with, because propagating parameter error into predictions is what a confidence interval is, you know, we'll cover, you know, ways to deal with this in more complex models. But that's the same, you know, it's the concept of what a confidence interval is, it comes from propagating parameter error. Okay, and that last bit, the errors in the processes, which can be both the do-do inherent stochasticities in systems, parameter error will decrease asymptotically, but the inherent stochasticity in a system may be something that's irreducible. Heterogeneity in systems may result irreducible uncertainties. Structural uncertainties in models, technically reducible, but every model is always an approximation of reality, no model is perfect. It will never be perfect, it's not supposed to be perfect, because if it was perfect, then it would be as complex as the real world. So there will always be structural uncertainty in models. I'll throw out that my personal hypothesis with some data behind it, but still to be determined, still to be borne out, is I actually think, when I think about these five terms, I think this one, the heterogeneity and variability, this statistical random effects, is actually going to be one that's going to dominate a lot of ecological forecasting problems. So if I were to put my money on it, I think ecology is a random effects forecasting problem, not an initial condition forecasting problem. It's something we all encounter when we work in the field. I study this plot, this watershed, this population, and then I got this question in prelims and it stumped me. If I map every tree in this whole watershed and someone on my committee asked, what about the next watershed over? Can you predict what's going on over there? It's like, I don't know. I have absolutely no idea after measuring every tree in this whole watershed, whether it has anything to do with what's going on in the next watershed. And I think one of the challenges in ecology is we're coming from a discipline that very much historically was focused on that very small scale heterogeneity, my plot, my watershed, my population, and have not thought, even struggling, one of the growth pains of ecology is struggling to think about how we scale this up, how we deal with the heterogeneity and variability. The fact that when I move over to the next watershed, yeah, the carrying capacity is a little bit different. Why? Well, we don't understand yet, but we need to accommodate those uncertainties, even if we can't explain them yet, because they will have a big impact on our forecasts. So here's a simple simulated experiment. So here are, envision 10 plots, populations, whatever your favorite thing you sample, measured over 10 time periods. So I'm going to call these sites in years. I've cooked up this example so that this set of time, 10 time series, and this set of time, 10 time series, have the exact same variance. So if all I'm looking at is residual unexplained variance, they are identical. But in this case, most of the variability is site-to-site, and in this case, most of the variability is year-to- year. I don't understand why there's site-to-site variability here. I might not understand why what's causing the year-to-year variability over here, but I can accommodate that variability, even if I can't yet explain what's causing it, and it impacts predictions. So if I am at this plot and trying to predict next year, I actually can predict it fairly well. By contrast, if I am in year 10, but trying to predict what's happening at a new site, I might have a lot of uncertainty about what's happening in a new site, because there's a lot of site-to-site variability they don't yet understand. By contrast, in this system, if I'm trying to predict a new year for a site I already know, I might not have much confidence in predicting a new year, but if I want to predict what's going on at a new site in this 10th year, I might actually be fairly confident. So again, the idea here random effects can account for accommodated, account for the variability, partition the variability that we can't yet explain, and it does have real impacts on predictions. That said, predicting new sites in new years are equally uncertain. You go back to this fact that they're identical. So here's just, again, in logistic model showing, for example, accommodating just additive process error at the end versus accounting for parameter variability, the idea that r and k themselves may be changing from year to year, and that would be a representation of variability in the process. Here it might be inter-annual variability in the process that we don't yet understand, versus just residual variability. So one take home from this is we take a simple model that we all were exposed to in our probably first ecology class, if not an intro bio class. It's a very simple model. I'm predicting n, population size given r and k, growth rate and carrying capacity, and I need some initial condition. So this model has three, one state variable and three parameters. Okay, well we just walked through, I needed to quantify 11 uncertainties to forecast that problem with one state variable and three parameters. So I guess one of the things I want to throw out as a take home here is that when you move from theoretical modeling, well just like it's been beat to death in theoretical modeling, to making a forecast with that model, all of your emphasis suddenly shifts to the uncertainties. That's where the bulk of the work actually ends up being done. So yeah, again the message, think when you think about forecasting, you're thinking about probability, you're thinking about distributions a lot. Last bit related to the theory was to point out in that overall term, simple version of the model of the derivation I showed that had five terms, I left out all the covariance terms, but they're in there. And the covariance terms are important and I think they could potentially tell us a bit about how ecological processes might scale and definitely how ecological forecasts should scale. So for example, as we move up in scale, we're going to average over the uncertainty in things like drivers and heterogeneity and process error. As we move up an average over variability those should become less important because they're becoming less uncertain. So things like internal stability might increase in importance. It also tells us that if you want to scale things up when we make forecasts, it's going to be very dependent upon those covariance terms. So the spatial and temporal auto and cross correlation. So that covariance term gets really important when you move up. And so one of the things I've been thinking a lot about is when I make, when I think about how forecast scales, so first and foremost, what is this covariance? So like, how does the information measured here decay in space as they move further away? So what's the rate at which information at one location is actually relevant further on? And then what might cause those correlations? So what, so if you think about it this way, if the world was simple, I could predict what's going on at this plot, predict what's going on at this plot, predict what's going on at that plot and they're all independent, in which case scaling is just summing up. So what causes that not to be the case is the fact that there are correlations between these different things. If there weren't correlations, then scaling would just be summing up. So what causes those correlations is key to thinking about how forecasting scales. So to come back to the Friedrichstein diagram and think about it from what we've just learned, we now can see that one of the things we see in this spread of ensembles is when we have one single projection for each model, we're confounding the structural error, the driver error, and the parameter error because each of these models themselves should have a confidence interval around it. We have an initial condition, present, we didn't use any data to constrain. These initial conditions were constrained by the assumption the world was in equilibrium in 1850, which it wasn't. There's no representation of process error, there's no representative heterogeneity and variability in the system in any of these projections. And if we added confidence intervals, which might be most important? We're trying to predict, in this case, you know, if I'm trying to predict the global carbon cycle, I don't know which of those five terms is driving the uncertainty. There's evidence that structure matters, but again, we can't say that for sure because this spread confounds the fact that the models have different parameters and the models have different drivers because they're coupled to different atmospheres. The other thing that I want to point out about this framework is it's not just a qualitative one that makes qualitative predictions, but it's also a quantitative framework that can be used to actually partition uncertainties in real forecasts. So this is a very simple forecast I made of carbon flux at the Willow Creek Flux Tower in Wisconsin using just a dynamic linear model, so no complexity here, just linear model of current flux plus temperature plus light, I think. But I can partition out that uncertainty in cumulative, in this case it's cumulative carbon flux, into the contribution of that process error, uncertainty in the driver's parameters and initial condition. So we see that initial condition uncertainty dropping exponentially. As we expected, we see the driver uncertainty increasing through time because the weather forecast gets more important. And we see the process error, since this is cumulative carbon flux, we see the process error declining over time because the random errors from every half hour start averaging out. So to come back and think about kind of reinforcing what I said earlier, we need as ecologists, one of the things we need as ecologists trying to think about making forecasts is to understand what type of forecast problems we have. And I think this is going to be really important because it allows us to think about, you know, from a theoretical perspective, what drives the dynamics of systems and how general are they across different processes and across different locations. It's also very practical. We need to be able to classify what sorts of problems are predictable so we know how to tackle new problems. If we can go, oh, well, we learned that this certain class of population models are all dominated by, you know, uncertainty X and therefore if we encounter a new problem that's like that, it's likely to be dominated by those sorts of uncertainties and we can focus our attention on those. One of the challenges in ecology and ecological forecasting is the temptation to say everything matters. Everything interacts with everything and therefore everything matters. But not everything matters equally. So we need to be able to say, if I'm encountering a new problem, where should I focus my efforts? What's likely to matter? Because I can't measure everything. If I know a certain type of problem is going to be initial condition dominated, I may make a very different set of measurements that if I know it's dominated by driver uncertainty or if I know it's dominated by site to site heterogeneity, how I make measurements and how I make forecasts, how I monitor that may be very different. And so that leads me to the last bit about what we measure, how we build our models, and how we assimilate data into models is ultimately going to be driven by which uncertainties dominate different classes of problems. Which brings us back to, you know, the actual theory behind the Nephi project. You know, I kind of laid out, we're forecasting a bunch of different stuff. Why are we forecasting a bunch of different stuff? Well, we're forecasting the, essentially the things that were as different as we could find in the Neon Data Catalog to make forecasts independent of each other. To ask the question, are there common patterns of predictability? So if I learn something about predictability in small mammals and ticks, does that tell me anything about my ability to predict harmful algal globes? Are there common patterns of predictability? And so actually for me, right now I would say in this project is one of the most exciting periods of my career because not only have I kind of grown to becoming just a broad ecologist, but I've done it because for the first time I actually see connections across ecology. I see that we are often working in individual silos and not always learning from each other. And that's because the way that we do work often doesn't make it clear how, you know, work that one of you is doing could actually affect our understanding of how other systems are working. And so, you know, the kind of what we, the place we got to, you know, what I call kind of the first principles derivation of predictability is that it left us with a place where because we don't have a single model that applies to all ecology, but we have a framework that applies to most ecological problems, we couldn't take that derivation further to the point to say, like in weather forecasts, where they could say, aha, it's always going to be initial conditions. Instead I think we've reached a point where understanding predictability is going to be a comparative problem. We need, we need enough literature, enough synthesis, enough examples of ecological forecasts to start understanding from a purely comparative perspective what is and is not predictable in ecological systems. And so we're trying to start building that library and start asking, do common things affect the predictability of different systems? And then quickly in the interest of getting you guys to a break, go through some of the methods, okay, we talked about the theory, we talked about predictability, so a little bit about the how. So what are the methods that we use to make forecasts? So I'm going to, a lot of what I'm going to focus on in this idea of iterative forecasting is the idea that forecasts should be updated when new data become available. So we make a prediction, new observations, we put things into the model, make a prediction, get a forecast, new observations become available, we want this ability to update those predictions every time new observations become available. The parts of this class that I teach, which are not all of it, are going to emphasize that to take a very Bayesian perspective, because Bayes' theorem has this nice inherent structure to it where you take your prior understanding of a system, you update it with new data and you get an updated understanding of the system conditioned on that new data. And then this current understanding of the system can then become the prior for our next round. So it's one of the reasons that I think Bayesian approaches are popular in ecological forecasting is because Bayes' theorem is itself an inherently iterative way of doing inference. It is always taking what we know and building on it. And so we're going to use that. That said, Barbara is going to come in tomorrow and give you a machine learning perspective that is very much deliberately brought in because that is the other perspective that is being used in ecology. And it's not how I think. And so I was like, I can't teach you that. I want someone who can. So to show you how I think about things I think, you know, here's my current uncertainty in the system. I make a forecast that uncertainty gets larger. I make some new observation of some new data with uncertainty. And then by Bayes' theorem I can combine the uncertainty in the forecast and the uncertainty in the data to update an updated estimate of the state system which I can then re-forecast from. High level take-homes, as I was saying earlier, we have to know the uncertainty in the model. We have to know the uncertainty in the data because the thing that we get out of data simulation essentially is that the precision of those two things controls the answer. If the model is imprecise, the forecast goes to the data. If the data is imprecise, the forecast goes to the model. The relative uncertainty. So if you fail to report uncertainties about a model or failed to report uncertainties about a data, it gives that thing too much weight. It makes it overconfident. The other thing that I think is really powerful about this data simulation approach is its way that it lets us combine observations to this idea of data fusion. So a simple example. Imagine that I have a model, and I'm very plant focused, where I'm making a prediction of the fraction conifer on the landscape and the above-ground biomass in the landscape. And let's say that the forecast of that forecast has a correlation. So forecasts that predicted higher conifers predicted more biomass, forecasts that predicted lower conifers had lower biomass. These blue distributions are just the marginal distributions here, so if I just take these points and make a density of them. So then let's say, imagine I make an observation of the fraction conifer on the landscape. That could come from remote sensing, inventory plots, you know, in our paleo work, it's coming from fossil pollen. I can combine that data with a forecast to get an updated estimate of the system, again using Bayes theorem. But the powerful thing is, because these two things are correlated, I can also update above-ground biomass in proportion to the strength of this correlation. So data simulation lets you update what we call latent indirectly observed variables based on the correlations in the forecast. And this works in both directions. If I observe above-ground biomass, I can update fraction conifer. If I observe both, then the constraint on each comes from the direct constraint from that observation and the indirect constraint by the things that it's correlated with. And this is actually kind of why weather forecasting works. So you think about the weather forecast, they need to initialize millions of grid, volume grid cells of the whole earth, you know, they pixelate the whole earth in three dimensions, and they need to know about a dozen state variables about every single pixel in the whole earth. Well, they can't possibly make direct measurements on all of those every six hours. They're inferring those a lot based on the correlations with other things that they can measure. So it's always, the weather forecasting is an under-specified problem. They don't make enough observations to actually say what is the state of the system. They have a lot of indirect inferences. And I think this is something that we potentially is a strength that we can leverage in ecological problems as well because I think in a lot of ecological problems, we have many data sources that each provide us partial information about the state of the system. And so I view model data simulation as kind of this problem. I call it as model as scaffold problem, where I have different observations about the state of the system. In this case, again, I think about the carbon cycle where I might have, you know, remote sensing at a kilometer resolution and fossil pollen at a 50 year resolution and flux towers at a 20 Hertz temporal resolution and, you know, leaf level measurements over two square centimeters. And I can't run a regression across any of these things because they operate on such different temporal and spatial scales, but that each tell me something about how the system works. But if I have a model that makes predictions across multiple spatial and temporal scales, I can use that model as a way of letting different data sources talk to each other. And in some sense, the model itself just becomes a giant covariance matrix. And it's really a data driven model of how we allow different observations to talk to each other. So to wrap things up, ecological forecasting, it's more than just running a model forward into the future. That's actually the easy part. It requires this fusion between models and data has to deal with multiple sources of uncertainty and variability. There's multiple. We've emphasized five key ones. And the importance of thinking probabilistically. Those uncertainties are important. I think a lot of what I end up doing when I work on any forecast problem is spending a lot of time thinking about probability distributions and how. I guess one way to think, I didn't say this earlier, thinking about the world probabilistically is important whether you believe the world itself is deterministic or not. So if you could believe the world is deterministic, but our understanding of that is incomplete, our knowledge of that is incomplete. So even if the world itself is fundamentally deterministic, we can use probability distributions to capture our current lack of the fact that we have incomplete knowledge and incomplete understanding. So for me, thinking probabilistically is very different than slapping stochasticity on a model. So this is not about stat slapping stochastic processes on a model. It's about using probability distributions to characterize and quantify what we do and don't understand about the world right now and always viewing that as a work in progress.