 So state space models, so again to remind you of the kind of inferential world we've been in you know We generally think about Modeling statistics as kind of signal versus noise and and we have some model of that signal and in this case you know a linear model With some kind of slope and intercept and we put biological or ecological relevance on those parameters often times that that we want to Assign signal to and then we describe the uncertainty in some ways that we've discussed yesterday We went through all the different kind of assumptions that we generally have when when thinking about this type of modeling approach where you're fitting some linear model through a cloud of data and Sometimes those data are missing and what do you do when they're missing you guys have now all done confidence intervals and predictive intervals And so you know that that you know You come out with confidence intervals that are based on how tightly you got those Parameter values so the precision of the parameter values right and you can always make those confidence intervals smaller by adding more data in You get better parameter value estimation but when you do your predictive interval you want to include some of that uncertainty because You're predicting so if you're predicting at a given spot here on the x-axis you would start with The mean right so your your prediction would be at the mean But you would have some interval around it based on both that observation error and how well you were able to characterize the model Okay, so this is kind of all background what we've done the last couple days And when we're forecasting we're generally thinking out beyond the end of when we have data So out of the range of the data that we have and so you can imagine that the the confidence intervals And certainly the the predictive intervals get bigger when you don't have data And that's again a standard approach you fit a line and you can predict out You assume that that line is representing a bigger population than what you've sampled So so if this is like some atmospheric measure and this is time then if we extended the types of Inference that that we've been talking about where you fit a linear model through a cloud of data And you try to predict out in time You can imagine that the things that that we've talked about the reasons that you have uncertainty in the data all come into play at once Right like so we're outside of the range of our data We still have observation error in often times certainly with climate or with a lot of ecological things We're also making predictions based on variables that themselves that aren't Perfect, they haven't been measured completely and maybe this this atmospheric measure is actually a compilation of multiple measures And so all of that makes makes this prediction issue difficult Ecologists have not had a ton of success in making forecasts or predictions that are then Validated in five to ten years Even when you do wait five to ten years and a lot of it, you know if you predict on this mean line Even if you predict within your data range You're almost never right if you were to make a prediction on the line and then go back and collect data You know, you don't necessarily expect to get Within you don't expect to hit the line in fact, you almost never expect to hit the line, right? And so understanding how off the line and why you might not be on the line is Super important and it only gets more important when you're trying to do it outside of the range of your data When you have a bunch of late variables, so again, you're you're measuring something like MPP and And yet you're really measuring something else But you're interpreting it as MPP and you have them connected in time So you want to know how that ecosystem measure changes over time Then you start to get into the realm of state-space models, right? So one thing I didn't point out in the last figures I put time on the x-axis and then showed you a linear model that obviously violates The idea of independence you could just jam all that into correlations in the air but That's not necessarily going to help if you're forecasting It may help you get a better p-value estimate or be more confident about your p-value on the slope But it's it's not necessarily going to help you forecast outside of your data better So when you have late variables that are connected in time or space That is they're not independent a State-space model is also a dynamic model. So it's a model that that a future state depends on the current state or And in much of what we're going to talk about It's going to depend the future state is going to be the next one, right? So it was just a lag of one, but but it could also be different lags If it's a linear process that that you're thinking about then it's also called a dynamic linear model So I'm putting all this up there because you may might hear some of these in ways that seem kind of interchangeable and And they often are But so dynamic model is State-space model and if it's got a linear process then dynamic linear model So when I say the the current state, right x of t depends on the The current state in this case would be t plus one depends on the last state So knowing something about what happened tells you about what it should be now state in this Vocabulary world is the the variable of interest right the the state of the system and if you remember three slides ago I Said it was latent variables connected, right? So so the other part of this is that that you have that data model we're out there met measuring y and then interpreting x right, so you have your process model, which is the f which is the X's So so now we have moved from X's as always representing your Predictor variable Okay, it is now a latent variable of Interest your response variable But not the one you measured and in this case I'm showing that that is where the the Markov process is right That's where you've got that Dependent structure one state depends on the last but your observations why are Still your data model they're what you measure and then you put a model in there to identify how they relate to X and Importantly you get process error and and Observation error right so you can partition the error out and and because this is going to be a Bayesian process You then have distributions on those errors that that you're going to estimate so the state space model is really It's just a really flexible framework for identifying how States of interest transition through time or space They don't need to be normal neither X nor Y needs to be normal. They don't need to be the same type of data You can have multiple types of data multiple Y's informing an X so yesterday I showed an example of fecundity that was informed by cones and seeds that were collected at different scales and sometimes in different years So that's two Y's informing X the fecundity They don't they don't need to have the same time scale missing data are fine again This is a probability base so everything unknown including an NA in your data set is treated as a random variable Right and it'll be drawn from that that data model and and a handles multiple data sources So it's a really powerful inflexible framework for forecasting and for ecological Infrared more broadly what we've been talking about up till now is Process where knowing the last state t-minus one tells you everything you need to know about the current state Plus error right so it's a random walk. You're you're basically adding a little error each time To kind of move away from what you were the last time right and so that's the same thing I I showed before but now I've Specified that this is a random walk model and you'll hell there that term come up a lot. It's used as a null model You know this model tells us that there's no there's no predictor variables in here, right? So there's no kind of explanatory process. It's a null model that Sometimes does better than your best hypothetical explanatory process But kind of in the gut really shouldn't if you got the explanatory process better So this is often where you start right you want to get this working first and again to break this apart again in the graphic notation We've got these X's these are our process model This is the true state not what we're actually observing but the true state in the example I did yesterday this would be fecundity. So this is like the the actual fecundity of a tree at the last time The current time and the next time Okay, you know, there's no more explanatory variables in this process. It's just knowing it tells you what it will be We have a parameter model and and you have some error that you specify and fit on that process Right, so each time you know where you are There's a little error that's added and that's how you get to the next time and there's no real structure to that error other than What you specify and the data model? These are what you observe and so importantly what I want you to get from this kind of slow tedious process of building this graphic Notation is that these are dependent on each other and these are not these are y's are independent Conditioned on the process and X Okay, so which is Everyone should go. Oh, it's a big deal, right because because you've got away from independence So that's you know, we're the error in the data model does not is not beholden to the Independence assumption you put your error on those y's as well, right, but it's not the holding to independent assumptions So what does this look like in code now that you guys have all had some some time with Jags and the code. We've got your data model. They should look really familiar, you know, your yi's are Normally distributed With some mean X That's your true state and the true state is normally distributed around the last true state and Then you have your priors because you're estimating both of the errors your process error and your observation error and There's a new line here your initial X. Yeah, so when you start you don't have an initial X You need to put a prior on that your initial condition. These are data that are New York cases of flu from 2015 We've got the months across the oh I should have said this is the cases of something in New York City in 2015 and the months are across the X-axis and what kind of epidemic Happens at this scale, but so it's flu and and there's some missing data here and they're kind of important missing data, right? because You can imagine if you're dealing with in the public health department or if you work in a hospital or selling Clinics is you know, you kind of want to know is this gonna like is this peaked out? Am I gonna sell a ton more Kleenex? Should I stock the shelves? There are reasons that that knowing these data are important. It's obviously non-linear, right and and it's also So it's the flu So one person with the flu gives the flu to you know two people who give the flu to four people, right? So you know that there's some some structure in the actual spread of flu, right? So none non-independent cases so obviously we're not going to fit a line Through these these data and and do anything useful And so state-space approach And so I ran the state-space model on that the random walk and in this case it did a pretty decent job coming up with converged Process and observation error. There's a lot of data here But this is one of the things so it it's one of the strengths of a state-space model is that you can partition observation and process error But it can also be really difficult to partition observation error and process error You can you can get real trade-offs Among the two, you know, and you could you could see some kind of bivariate Densities come out you In general if you have any way to constrain Usually observation error if you know something about how off you might be and you can constrain Observation error, then you can get a better estimate of process error And so this is the so this you know the the random walk was run and Things converged yay, and then you make the credible intervals around that so these are the 95% credible intervals you know and surprisingly you didn't do a great job at the at the initial state didn't okay job at the late state and You have the biggest uncertainty where you don't have data And in fact that uncertainty gets bigger the further away you are from data, right? If you remember the the graphic notation best information the best estimates You're going to get about xt are always going to be where you have t plus 1 and t minus 1 If you don't have either of those then you're going to get less and less certain about xt Which should make intuitive sense And so that's what happens here, right? You're you're not super certain And and you know you get some information the the flu could peak at 3,000 cases or 6,000 cases It's kind of what it's telling you and that that's better than no information and It may be less certain than if I had done the model Without characterizing the uncertainty and propagating it, but that doesn't make It wrong Right, you're let you're more likely to actually Have the the true estimate be within that range Even if it may seem less satisfying to have a bigger potential range. It's the right bigger potential range I'm gonna you of course this time you have the data and you put it back in and yay I did a good job And it actually looks almost kind of like the air is symmetrical but there are a lot of data and So knowing last month's flu turns out to be a pretty good predictor of next month's flu But what if you only have the first half or you don't even know if it's the first half What if you only have like initial ramp up of the flu and you know you've run out of vaccines And you want to know if maybe you've started to make a difference or things are actually going to get worse Right, so again, this is an important question and right now you've got lots of data in the t-minus one kind of Realm but as you get to the t-plus one at realm Plus one realm you're getting into places where you're going to have to be estimating a bunch of x's with no future information And so again You would expect the uncertainty as you get further away from here to get bigger It doesn't matter if you you know can can fit a spline or something to this perfectly And you just extend it out You should still get less certain about where you are as you get further away from the last data point you had Okay, so hopefully intuitively that should make sense and in fact you can plot that out and even though The first model where you had some information back here to constrain it again look like you know knowing last month is a really you know Powerful way to predict this month. That's only true up to a point Right, so this this is good, right? There's more information out there. We're not completely helpless Beyond knowing the current state as a predictor of the future state. There's got to be more information out there that is Important so this is a random walk, right? And and even though it looks like it does really good It only does really good if you have a lot of data and you have data You know on both sides of the series you're trying to to forecast and oftentimes We're not trying to forecast things. We already have some future data kind of to bound so, you know, how do we go from boundless possibilities in in the future to to actually putting some bounds on that which I Mean this is almost useless Right this forecast here is almost useless for for management of any sort And obviously you would want to pull out some predictor variables that that help constrain that beyond the random Null model and so a dynamic linear state space model is you actually just Kind of add the linear model in right so that the true state is still a function of the last true state but also of a linear process and There are many Special cases where you move beyond the linear process and then also one today But you've all been doing this and in this case I'm putting temperature in here Which doesn't have there's some biological reason to think temperature is predictive of flu But mostly I chose it because things peak in the winter and and then you still have your data model. I didn't model this But you can Imagine that there were temperatures that were lower in the beginning of the flu season and as they got lower again The temperature should have some ability to pull that uncertainty air in right okay And there are lots of other things you would want to put in if you were doing this For some public health reason So so you can also do a non-linear state space and that the example I'm going to cover here is kind of a mercury capture example I'm a lot of people here who think about animals and and how you sample animals and in this Sampling mechanism you go out and you capture animals and you tag them and then you let them go You go out again and you capture them again, and you assume that the proportion of animals you recapture, but you assume that it's some constant some Proportion of the total population Right you marked some proportion the first time and when you go back out and you recapture and you get those marked ones back that Right, it's a ratio you get the the proportion of marked ones you get of the ones you marked is proportional to the proportion of the Initial capture and the total population Okay, so the more you do that Presumably the more information you get we there are some assumptions. It's random, and it's not a hundred percent Which usually isn't a hard assumption to meet Obviously don't do it with plants. Okay, so the data look very different than than the last flu counts So you an individual record the individual is I and the record itself is a vector of In this case somebody went out five times and they captured it then they didn't then they captured it And they didn't and they didn't and so there are different Realities that match with this capture history. This is a why and observed capture history This is these are the potential true states So here's Z is the true state and the potential true states are that it was alive here and And pretty much that's the only one that works there because then it was alive again Right so all of those true states involve being alive, but not captured It because then it's captured again, and you kind of assume if you capture it that is Unless the the the way you marked it was in some way vague If you capture it and it's been marked it's alive and it survived until you captured it right by definition But you don't know about these last two, right? You know that that your capture probability isn't 100% because you got the zero and then it showed up alive again And that actually is good. So you don't know about these two So you so the different possibilities are that it that it died and you didn't capture it or that it survived one more capture One more sampling period or it survived all of the sampling periods, right? And so now we have a true state that instead of being a continuous distribution We're not going to describe it that way. We're actually going to define the discrete possible states and assign them probabilities From this we we don't know when it died But we do know that that we didn't capture it with a hundred percent probability And so we get some information about survival without being recaptured So this is the graphic notation. You've got your your observations that are independent, right? There's no arrows connecting them You've got your process model that are dependent through time and you've got your in this case The parameter model is doing error that also varies temporarily You could just have one P and one S and you could say that all X's have the same constant variance Or you could imagine that variance might change throughout the season and Your ability to observe might change throughout the season. So in this kind of Example we're assuming that the variance in our ability to observe given that it's alive and our variance in being alive vary during the course of T's during the progression of of these data So survival we're going to use a Bernoulli Probability and in this case again, we're setting up these these kind of discrete probability Descriptions so you've got the probability that X the true state at time right now is alive Given that it was alive last time Okay, so we're defining that as ST and the probability that's alive giving that it was not alive last time is zero and Then the probability that it's that you don't see it given that it was alive last time is just one minus That alternative right there are two options there and and and they add up to one and again There are two options here the given that if it wasn't alive last time. It's either Not alive this time or it was like last time. We already said that's zero. So it's one, right? So you've got two different ways that you're getting to a probability of one The probability of your different states, which are zero and one each have to equal one because we're in probability realm So it has to equal one and Then you do the same thing with the observation model But the observation model remember is not Why is aren't dependent on each other? So it's why is not alive condition or you don't observe it conditioned on the fact that you really Didn't estimate that it was alive last time and then you put your priors on the errors in this model And and I I guess I will point out that I I've noticed as I've gone through these slides sometimes I'm using t plus one and t and sometimes I'm using t and t minus one Just remember the graphic notation where you have x t is in the middle and they're both important So if it wasn't bothering anyone to forget I just said it And so this is actually data from her own John Foster who is working on mouse recapture study that that's 11 years. So so I wanted to have a Final kind of figure to show that you know even with this non-linear Approach you you get very similar credible intervals and they get wider, right? So, you know these parts Are we put the data on but those parts of where where there aren't data and he's estimating this on a daily You know, are they alive on a given day? and so there's lots of days often in between sampling bouts and the longer that goes the wider the uncertainty gets which is what you would expect and and these Solid lines are the minimum number alive. So, you know that that you caught ten There's at least ten out there that are alive And so we plot that to make sure it dip and below that which would be an indication that we had the Dead things showing up in the trap. Yeah, but it is also just really cool data. So he's doing a good job of it And so that's the end of the kind of introduction to state space That I had prepared