 Okay, let's let's start with a little bit of conceptual stuff So this is a device for testing colorblindness. Anyone here colorblind? Yeah, okay, what for? Okay, well here we go. Here's a great Somebody every class it's all you guys you guys normal Yeah, exactly what guys exactly So these these tests are quite common and I was using this one with my son. That's why there are animals here, right? So you could say find the animals and go through it. I knew my son was a colorblind because the medallion next makes it possible, but But it was a fun thing to do and he thought it was interesting so So yeah, there there are some animals here. There's a bear up there, right or a Bruin rather I would call it at least one beer Deer and a cow superimposed right exactly the next in the purple and Yeah, there's various things some bunnies and squirrels and Yeah I Think et's in the corner Yeah, well anyway, so what's the point I want to make with this? This is a test right and it's extremely diagnostic It has almost no failure rate if someone is colorblind they can't see stuff here The hues and the intensities the saturations are planned out, right? So that the shapes become invisible people with different forms of colorblindness and with this one diagram you can diagnose I think three different common forms of colorblindness and So, you know the bear is invisible to Josh and Mostly invisible and and it's very diagnostic. You can't fake it. You can't find the bear There's just no way to do it. And if you if you don't have clothes line, it's the bear is there How can you not see the bear? There are phenomena in the world that are like this where there are easy tests that produce discrete outcomes And there's no uncertainty about the outcome about it. Some things were like this Unfortunately, most of our science isn't And yet there's this real psychological need to develop diagnostic Discrete tests with binary outcomes and I think a lot of the the cultural evolution of statistics in the sciences themselves Not so much in stats departments is about fulfilling that wish It's the wish fulfillment of having basically colorblindness tests or whether your hypothesis is true or false so This is I put this in this part of the course to sort of say well look we have to talk about the values and The reason is it because this is a Bayesian course because you can do Significance testing with posterior probability just as well as you can do it with frequent just sampling distributions You can people do that Bayesian psychology in particular are really big on the hypothesis testing with Bayes factors is what they say I'm down on all that stuff and that's why I don't teach it. It's not a Bayesian attitude It's instead. It's the recognition that but scientific evidence is probabilistic It's not like colorblindness where it comes out discreet like that, but there's some threshold and yeah You pass the test or you don't evidence is massively uncertain what we have a probability distribution and In all statistical frameworks Bayesian or not probability encodes uncertainty It uses a different metaphors to do it But that's what it always does and translating that probability into certainty is something you can only do through some other routine It's not part of the machine itself, right? It's a heuristic that's imposed on it So that's why I don't I don't teach this because I don't want you encourage you to do that I want to encourage you to embrace the uncertainty give it a warm hug get used to it and Communicate as much as it is you can to your colleagues So that they can build on it honestly right instead of losing uncertainty at every step So let me run through I think this is it's useful to do before we get into material today run through all the reasons that P values are basically the worst tool you can use for whatever you want to do and So if you're interested in estimating an effect Deciding whether the effect is real or not based upon the p-value is worse than using the maximum likelihood estimator Which would be like the map estimator with black priors or just using the full posterior Which is what I've been advising you to do. That's the estimate Additional on the model and data the p-value is some additional procedure Where you're deciding and people do this all the time that there's really no effect that is the estimate is zero Because it's arbitrary threshold is on the past right and five percent is magical thinking right there's nothing about five percent that matters Because there was this fish that crawled out of the sea that had five on the rays It stands and that's pretty much it. I think that's the true story actually we can we can blame lung fish for the five percent convention and damn fish and So if you're interested in estimation Look at the posterior distribution or if you're doing frequentist estimation look at the sampling distribution That's what you care about But yet that's the accuracy issue conditional on the model If you're interested in deciding How you should behave based upon an estimate then things get interesting But there's this whole field of decision analysis which was mentioned in in chapter two right or what's chapter three now? I forget I think it was chapter three Yeah, it was the end of chapter three and I didn't have time to talk about it much in lecture where in order to do this You need to you need to decide what the costs and benefits of different kinds of mistakes are now would be a formal decision analysis And then you could decide how you should behave Initial law of certain estimates or a certain posterior distribution and there's a big interesting field about that So in this context the thing about using p-values to make these decisions about whether we're going to behave as if this is a Hypothesis is correct or not is that it's uncalibrated always There are no explicit costs and benefits and the 5% is exhaustionally determined in arbitrary because of a low fish Right, so I don't think there's anything wrong with doing decision analysis But you should be explicit about it and realize that when you're deciding to act as if the effect is actually zero You're doing an extremely naive form of decision analysis, right? No one to talk to you that way so can't blame yourself You shouldn't blame yourself. There's a tradition here that's evolved in the sciences as wish fulfillment Do you want to do prediction? Well, we did all that stuff last week, right? I probably don't have to reiterate this there's nothing about 5% was optimizes anything guarantees anything there are Different and successful prediction frameworks. I taught you one based on information theory. There are others as well Those are justifiable in a way that he isn't P is not the P value Procedure is not derived to optimize prediction and the 5% is definitely not derived to optimize anything Right, it's just an arbitrary convention So all this applies to confidence intervals as well And I have to say this because in you know, I think in psychology right now that this idea But they call the new statistics is using confidence intervals instead of p-values But they still look at the end like if the confidence interval includes zero no effect, right? It's the same thing Don't do that either. It's just a different language for describing the same stuff So don't give in to that temptation as well boundaries aren't what's interesting the continuous change in plausibility is what's interesting If that makes sense, so you kind of ease into this as we go. Let me give you an example and This is one of my favorite ones of this. I like it so much. I tweeted it this week. I think it's first time I've used that verb in public And This is a great paper for 2004 by how are and How are is a statistician of? Largely traffic safety, which is a big deal. So this is cars are the main threat to human life in North America Not so much in Davis. Here's bicycles, but So this is a great paper because what he does is he goes through three different historical cases of Hypothetical questions about changing traffic law in cases where accepting the null hypothesis because he was greater than point zero five led to increased human mortality And and the problem in all cases was the studies had low power another way to think about that is the posterior distribution was very broad And but it was broad enough to include zero but also really big values were plausible, too But people ignore the big side from your perspective. You're zero, right with the paper around your big values So let me give you table one from this paper is great Now the first story focuses on right turn on red, which is the same we do in North America that it done almost nowhere else in the world Why because it's crazy Crazy, this is how pedestrians die is that we make it legal for cars to take a right turn on red whenever they feel like It is a bad idea. So New York City is saying right because New York City doesn't allow this is that right Paul and so It used to be illegal in all of North America and then during the The oil embargo during Carter's pregnancy, I believe There was all this pressure to say well, are these people sitting idling at red lights What if we let them take a turn to use up that fuel open gas prices spiked and people did crazy stuff about it And so various states and counties started exploring with the idea of allowing right turns on red so there was less idling at stop signs and You can see here some of the count data that was that they came out of this You compare the same intersections before and after allowing right turns on red in different categories zero fatal crashes to zero fatal crashes 43 personal injuries before to 60 after 69 person injured persons injured before 72 after in every case the bad stuff goes up But it's a short period of study and none of these differences are statistically significant at the arbitrary low-fish threshold trying to fuck percent and so study after study in North America because none of them were cooperating it was each little county doing it by themselves and low power and all of them concluded it right turn on red was great and Then they made it law and now we're stuck with it and we're killing pedestrians every day in this country You're welcome for this sad story Class used to be fun I've got a bunch of these I have like the file of statistical horrors But so there are two more I encourage you take a look at this paper. Just just googled a how our accident analysis and prevention And right turn on red you'll find it two other stories to be one with speed limits and The other one with curves. I think putting in curves. Yeah fun. Oh, these are these were like month Trials and a lot of it is rural counties for trying these and you got a rural intersection and a month-long trial And in every case the estimate is in the direction of saying it's more dangerous, but the confidence interval was wise And these differences actually underestimate the harm in the long run as they're missing the occasional actual make will pay tally Which arises so it's again to think about confidence intervals is just because it includes you're on one end It noticed that it also includes really big values on the other end And the model is saying they're equally plausible So yeah, you have to pay attention to both sides the history. This is a great paper. I think it's it's really a jewel Okay so What should you do? Well guys know I've been about Horoscopic advice, but that's the best I could sort of do in in the class of this kind Here are things that I think I can get behind for ordinary sorts of regression Models the kinds we've looked at so far and I call these heuristics They're just rules of thumb when your domain knowledge is be something else do something else, you know your science and You can almost always beat my big advice Instead of asking whether or not the effect is real assume. It's real and estimate that's what your model actually does So you can address it that way and then after the fact is a separate question How should you behave conditional on that in basic science? Usually we just like to construct a posterior distribution and pass it on to others So others can build so for example in the right turn on red literature if they had done that instead of recording zero is the effect Which some of the papers did Which wasn't the difference the difference is not zero in that table right, but that's what they said there is no effect There was an effect Just because P is greater than five percent doesn't mean there that the estimate changed the estimate was still greater than zero If you communicate that you pile up the evidence if all of those papers that found those significant difference had just presented The posterior distribution all of them could have fooled it in the meta-analysis and found out that wow across The country there's massive evidence that right turn on red is dangerous and should not be allowed similar thing happened with speed limit increases in the 80s bunch of states were allowed to increase the speed limits and and they did the short-term low-power studies and it seemed fine But they'd aggregated them They would have found out it was a terrible idea and we're still we still have high-speed limits actually so Summon effect and estimate it If you must make a decision decision analysis is a thing you can do think about the New York Blizzard again, right? They got criticized for not being accurate, but accuracy wasn't the issue there But they were trying to protect the public and so they had to overreact Maybe they over overreacted they probably didn't need to shut down the subways New York subways didn't withstand the nuclear explosion probably everything else would be level, but the subways would still be running man by rats So it's a great subway system Aside from the spreading epidemic that some of you know what I mean so With in the context of any particular model imagine how the estimates can mislead you they're always missing variables You may have thought about and if you have good enough a domain knowledge about your system You may be able to think about what they are And maybe you couldn't measure them or maybe you didn't think of them ahead of time either way Make a proper accounting of how if you had those variables and if the world were different How do you think it would affect your results and you can often do if you have good scientific knowledge your system? You can do better than just anxiety here You can make predictions about the direction it should change things so sometimes you can figure out for example Okay, so we didn't measure the following variable But but any if any bias induced by that variable would be in favor of my hypothesis So my results are conservative, right? This is what you want to be able to argue Sometimes it'd be the other way and then you just have to be honest and say yeah, that other variable is important and untrue That's just how it is but Think about how you got your sample remember the manatees and bombers When things become a sample typically there are subsets of the whole population of the phenomenon So I was I was in a seminar yesterday where I brought up this selection effect in terms of graduate student emissions Which is a conversation I have every year about this time with people so it turns out in admitted grad students and PhD programs and and actually undergrad There's a negative correlation between GPA and standardized testing scores Nearly always and in many places many countries. It's not just a North American effect. It's very strong in PhD students And so people will say oh this tells us something about the nature of intelligence or they're good at coursework or good at standardized tests whatever that means and It doesn't it's a side-effect of selection So the way you get admitted to graduate school is that the sum of those things passes the threshold This is basically how it goes doesn't have to be a perfect song But there's some largely additive function that produces a score and if that score passes a threshold you can get into graduate school The grad studies here actually provide thresholds at which they will allow programs to admit to so this is a rule and As a consequence of that there There are very few people who are excellent at both of those things and really top them both out So the most the easiest way to get in is to be excellent at one of them And that creates a negative correlation in the selected sample But in the population as a whole they're positively correlated Right, does that make sense? This is a fantastic thing. It's called Beckman's paradox in statistics It comes up all the time the general lesson to take away here It's just think about how you got your data and and what cases might have been missed Could values of predictors have influenced the probability that data ended up in your sample If it can you can model that it's not the end of the world. This is the standard thing in polling Right, who answers telephones? boring people and nothing better to do And I only answer phone calls from my spouse that's the only person I will ever answer phone calls from literally so maybe eventually my son but So there's this problem in polling and dealing with selection bias polling samples are predominantly white and old and Those people answer phones more and have landlines. So you have to do sample adjustments deal with this and it's a big topic In polling in the statistics of polling lots of things have this is not the end of the world But you need to think about it. So again, the bombers are like that, too That's a system problem you can deal with and you imagine that the probability of bombers in your sample Well, it must not have been shot down And that flips the meaningfulness of a lot of things about the location of damage Manatees likewise right be able to see the manatee. It needs to have not been killed Model structure all your inferences are conditional a model structure imagine different structures and for now This just means different linear models But as we move along, I think you'll be able to see there are other things to think about you I'll have an example later today a verbal one I want to throw at you but leaving the mathematics aside for the moment of a case where these linear models aren't really capable of describing a Phenomenon you already understand So you need to expand beyond it Good rule to always keep in mind fitting is easy prediction is hard right Right now my sometimes it might seem like fitting is hard because you know your computer fights you right to get those scary red messages and are something about being in finite difference value my favorite one and So it seems like it's hard, but this is actually the easiest part It's easy to encrypt a sample of data using a model into a different set of numbers called a posterior distribution That is actually the easy part of this job robot stupid predicting the future is tough both because Models overfit that's the first problem. We spent a lot of time last week on that Secondly because models are never correct in the sense that the real natural processes generating data are always different from our schematics the models that we use And the future the phenomenon we're studying may be changing it may not be uniform Again in the social sciences and biological sciences It's easy to imagine that because we work at a very high level of description in population biology Many of you are population biologists So the the nature of the phenomenon is actually moving out from under your feet And you might have gotten the model trained correctly the first time but then Damn it the climate change Right everything changes now You know or or rats in bagers field site or whatever it is Plotting is is really essential and I think under emphasize in introductory statistics in advanced statistics is not It's emphasized a great deal, but in introductory statistics There's this over emphasis on reading tables of coefficients And I hope I convinced you guys by now I board you to death with it because it's like a mantra with me that it's hard to understand a model from tables of Coefficient so there's just marginal posterior distributions There's a lot of information missing what you need to look at are the implied predictions And that will automatically take account of all the co-variance among those parameters to help you understand things We're going to spend some more time on that today as I think it's very important now with interactions Plotting is the only way I can make sense out of them at least it's very difficult otherwise And then above all embrace uncertainties. There's a certainty about parameters There's a certainty about models report as much of it as you can to your colleagues so they can build Like you don't need to be overconfident if science goes the way I hope it does This sort of thing will become compulsory because we'll make everything Easy to replicate right with this reproducibility movement is a fantastic thing that's going on when I started grad school no one was talking about making their work reproducible and The idea that you would like send in a script with your paper the first time I did that the editor was like I don't want this And like scolded me was sending him my data analysis script now their journals that require this is a wonderful thing and With that I think the idea of being overconfident will fade because you can get called on it really easy Right and the incentive structure is going to shift. That's my hope I'm usually wrong though, so Maybe maybe I should hope for the other thing Okay We're in week five. We're going to move into week six mcmc next week And so you've you've got in addition to your homework For the chapter we're going to finish today You your other homework if you haven't already is to install stan Here's stan You're all installed the software named after him. This is a stanislaw ulam Very famous figure in the history of machine learning and also in in biology Which is why he's standing by this this organic molecule Here did lots of good stuff And so stan is a A programming language that helps you do statistical inference You're not going to be writing in it directly the rethinking package hides the guts from you until you want it And later on you can graduate to writing in it yourself. It looks a lot like the model code you're writing already though But with more precision So we're going to use stan is our markup chain Monte Carlo engine so we can fit more Kinds of models. We're going to be using this in the second half of the course to do generalized linear And generalized linear mixed models So get this installed go to the mc dash stan.org find our stan quick start here go there find your platform Get installed. I'm confident you can do it. You guys be more complicated all the time, but Keep it going. There will be some problem and then I will help you right We'll figure it out. But this is what you're going to need to do starting next week to do the homework. Okay and In rethinking we'll kind of fire it up for you once you've got it all installed The hardest part as I did it here is to get a c++ compiler installed If you're if you're among the blessed and you are on a unix platform already you probably already have one But max don't come anymore right with the compiler installed. That's right. And so you you've got to download xcode This will tell you how to do it when you go to their quick start drive. They'll tell you about that It's pretty easy to get free. You don't have to buy anything And if you're using ubuntu or something you've already got what you need like that And on windows, you'll probably need a compiler They'll they'll give you a link to it as well. That's the hard part and then the rest is easy Okay, but even if there are problems and there's always somebody who's whose computer's got a gremlin It's not your fault Don't engage in self-ladulation I will help you with it. Okay, and paul is also very clever with computers out here He denies any skill at this anyway do this. This is part of your side for the weekend. Okay Let's get back into interactions now. So Uh just to remind you where we're we're going to work on the tulip data as a clean example of an interaction where you've had the biological background to Understand what's happening and you'll be able to understand the estimates that arise Blooms We're gonna these are greenhouse replicates We have three values of water treatment three values of shade treatment these things interact because you know plants Right, they need both photosynthesis requires light and water and either is sufficient by itself Um We're gonna we consider two models the ordinary no interaction model where we just have main effects of water and shade and the interaction model Where we multiply the water and shade levels together. So these are continuous interactions. These aren't on off this isn't like the easy to think through um ruggedness in africa Thing that I started with that one because what makes it conceptually easy is that africa is just an on-off switch, right? It's a dummy variable. So you've got two slopes and the model estimates exactly two slopes In a continuous interaction like this It estimates a continuous change in slope as the product of the of water and shade changes And so suddenly this is craziness, right? It's hard to see the equation looks the same Constructed the same way, but it's much harder to think through now and we're going to need plots. Absolutely to do this Um So I think we this is where we had gotten to we compare the two models. Just show you the interaction model. There's a lot of evidence It's better Um, it fits the sample a whole lot better But even accounting for the the flexibility of the model the increased flexibility to model It's still the expected out of sample deviants is a lot better as well And what I've gotten to show you was uh, if you compare the coefficients of the non interaction model to the interaction model They change a lot. Uh, this is very confusing and this happens to a lot of people And uh, uh, it's normal to be confused by what has happened here You can interpret these coefficients, right? You guys are pros of these these simple main effect models. You can do it Each of these beta coefficients being in the in model 7.6 Just being the change in the outcome for unit change in that predictor, right? And the model assumes they don't interact so you can read them pure and think about prediction that way You can get good at that. So that's fine. So that's the intro stats reading table that can be done The interaction model it's all off Everything is gone crazy Now that this effect size is doubled it hasn't This one is flip sign. It hasn't really shade is still bad for plants. Okay in this model And then you've got an interaction effect and it's negative And what does that mean because this thing is negative or multiplying it by something and then it's getting added to a bunch of other stuff You can't think through this stuff. Just don't even try Instead we're going to make plots so First thing to talk about though is this is come back to the topic of the center And show you that if you center your predictors, you can make a little bit more progress on thinking about the coefficient tables I do want to show you this even though in general I want to do it just to kind of explain to you the key point which is the reason The strange changes in estimates on the previous slide arise because these parameters even though they have the same labels They don't mean the same thing anymore The interaction model is different. It has a different probability space a different model structure The fact that you reuse the same labels Uh in the model because they're conceptually kind of similar doesn't mean they refer to the same exact things And they can't reinterpret it in the same way So I want to walk you through that try and bring it home and then we'll talk about plotting Which is the way I really encourage you to figure this stuff out Um, so to remind you centering means we take each Predictor variable and we just subtract the mean for making each value if you also divided by standard dba should you have standardized ones But I'm not standardizing here to show you that the centering is what does the thing here? You can also standardize if you want. I think that's great. It'll make fitting a lot more reliable here Uh, but you don't have to do that Uh, the centering is what causes the effect So re-center those refit the models the two models the main effect model and the interaction model now using the centered predictor variables So think about what centering has done before we look at the graphs where you see exactly what it's done Shade had three values one two three Since it's an experiment. It's all balanced. So the mean was two So we subtract two from every value and now the values are minus one zero one Right, so these are the shade levels the same for water That's all it's done is just subtracted two from each of the values of the predictors Now we refit the models. We do the coefficient table again Intercept stays about the same and in this case the intercept means the average This is the average bloom when all the predictors are at their average values 129 is the average bloom area when all the predictors are at their average values, which are zero Right, that's what you get out of it. So the intercept is interpretable now. Some people get excited about this I don't because I just thought stuff anyway, but uh, that's fine The main effects stay the same almost identical right almost identical estimates, uh, within, you know, government work and uh We don't care about the residual variation anyway except that, you know, this is smaller because it's fitting the sample better Right and the interaction is negative. We're going to spend some time talking about that I think this is still hard to understand in any verbal way. So we're going to still need to plot that But this is what centering gets you is it it makes the main effects behave in a way They won't always be exactly the same as in the beginning here just because everything's balanced That's why they're they're so similar But you'll get these these effects. So let me unpack for you why this happened a little bit And the first thing is let's do a little bit of thinking about the structure of a linear model with an interaction Here's just the likelihood of this interaction model And you can think about the usual question is We ask these sort of regression questions What's the rate of change in the outcome per unit change in a predictor variable and in a simple linear regression? There's a parameter for that But in an interaction model there isn't I'll say that again in a in a simple linear regression with only main effects There is a parameter a beta coefficient which answers this question What's the change in the outcome per unit change of the predictor or in this case? What's the change in bloom size per unit change in water level? In a linear regression, there's a coefficient for that that is exactly that answer here There isn't but we can still calculate it how well that has a definition and this definition is a derivative And I know Tez has raised his eyebrow Riley because he loves derivatives Sorry, you did the eyebrow thing man. It's on you No, we used to joke he took my class last quarter. We joked about calculus a lot, but that's why it's an inside joke. So Now it's outside Exactly So, uh, we can calculate the rate of change in mu, which is the expected value of the outcome per unit change in w just by calculating this partial derivative and chain rules all you need here and It is exactly b sub w plus b sub w s s which is by no coincidence that gamma thing from the example on tuesday It's what we defined as the rate of change when we change that predictor that it depends It's a model itself. It's a linear model itself that creates the dependency So the other predictor is in this and that's what creates the interaction effect Does it make some sense? We've recovered our assumption This is what I love about math. It's all tautology, but you still learn stuff from it. I keep saying this, but it's true um And likewise for the other one the change in in blooms per unit change in shade We can calculate that one in unsurprisingly similarly, it's it's b s plus Yes b s sorry beta s It's avoid that particular linguistic risk. It is beta s plus beta w s times w And uh, so again a linear model two parameters in both cases are needed to answer the question plus some assumption About the value of the other predictor variable. And that's why it's weird So these these parameters don't mean the same thing they used to you know in a simple linear regression The answers to these questions are just the parameters beta w and beta s right Now the answer to these questions, which are what we want to know is this model So how do you deal with this you plot it? That's what you do And there's just no way around it if you want to think about it coherently now But let's let's say so why does centering help or appear to help? The meaning of the parameters has changed when you add the interaction and let me unpack what you learned on the previous slide into this explanation And centering basically disguises the change in identity by making the mean value of all the predictors zero So that it cancels all those parameters And then you're looking at changes at the mean But it's still just a disguise the parameters still don't mean the main thing the same thing And so that's why I think it's dangerous to encourage you to say like oh send me your predictors and then it's all fine It's not all fine. It's still a trap. Okay, it's a sexual trap and you want to you want to plot so Think about this in a coefficient model without interactions the change in the outcome The coefficient is a change in the outcome per unit change in predictor So like the change in blues per unit change in water There's a coefficient for that in a simple model that interaction that same parameter with the same label In an interaction model is the change in the outcome per unit change in predictor when the other predictor is zero Because then that little linear model b sub s Plus b sub w s times Um water The only time that will be only that one coefficient is when the predictor is zero So when you center you make the mean zero And then you get an inference at zero Right an inference at the mean about the meaning of it And so ends up being in the same sort of value, but it's just a trick really it disguises the change in identity It's still a different entity And uh Yeah, so let's say that that in a scent with centered predictors the mean is zero You coerced it to be that and that's how it arises. So I've gone through all this just to say Centering is great. It helps you fit the model better It'll lead you to be a little less surprised by wild swings and parameter estimates that you transition to an interaction model But still to really understand What the model says you're going to have to always use more than one parameter at a time It's just the truth of how these things go You can make inferences add a particular value But you want to make inferences across changes in values and to do that well We're going to have to do some plotting So let's take a look uh last time to look at one of these tables. Here are the actual This is the full marginal posteriors for the interaction model. Um m 7.9 We can think about the identity disease things now and you can see sort of how they're not really answering the questions you want Alpha is the mean blooms when water and shade are equal to zero Which here means at their average value So that makes this an interpretable value 129. That's a real thing in the uncentered one You will have forgotten, but it was alpha was negative, right? Which means it's like a black hole tulip that's sucking in tulips for the future something like that like it owes tulips to the to the future uh and Um b w the change in blooms per unit change in water only when shade equals zero So then the first thing I said, well, what if it doesn't equal zero? Okay, well, then you've got to use more than one parameter and deal with the covariance among Right to get the confidence interval of it Uh, and that's not an easy thing to do in your head and you can there's not enough information displayed in this table to do it Uh b s same thing change in blooms per unit change in shade when water equals zero Bws is the interaction. It's negative. What does that mean? I encourage you not even to try there There are various metaphors you can apply and try to understand it But I think they're very system specific and different contexts. You can get it Um, so the but I'm trying to give you a horoscopic advice in that case It's plot. So let's let's transition to that talking about plotting I'm fond of a of a technique called tryptics, which I learned in photography class as an undergraduate And the tryptic is a classic art presentation where you have three related panels images that as a whole create a more satisfying Very common in modern photography. Here's my favorite historical tryptic This is lewis pal who's one of the co-conspirators for the assassination of abraham lincoln And he didn't pull a trigger, but uh, he was convicted of conspiracy which he admitted to Because he was a proud southerner fighting the war of northern regression And uh, this these are great photos. You can sort of see the insolence and and pride in this This was shortly before he was awaiting his hanging Which came like half an hour after these photos were taken This is a proud young man who felt he was fighting for the rights of his nation Right, it's it's these are really interesting photos. Now, of course, it's dark Great because he was fighting for something we regard as i mean, he was a terrorist, right? He was an american terrorist These are great photos and you get more from the tryptic, right? You learn stuff about him And his attitude towards the photographer to the middle one's great where he looks at the photographer and is like come on Come at me, bro So anyway, so now art class aside We're gonna do this with data we're gonna do this with interaction plans and the value of the tryptic will be You'll get to see the change across channels and the whole will give you a more aesthetic appreciation about how the model behaves There will be no american terrorists involved, but uh, uh, you will learn a lot from it. So um Let's look at this uh this way. I wanted to say too. There's nothing special about using three It's just I think you need at least three so you can do two extremes and an average But you might want to do 20 if you like or if you're really good at fancy animated interactive stuff make something with a slider And That'll help you too But the comparison is often good. So more of the one panel is nice. So here's how you do it I'm not going to show you the code to do this because really All you do with the code here is we've done this before you fix One of the predictors at a certain value and then you generate predictions For the cross values of the predictor on the horizontal you've been doing this over and over again But now we're going to choose different values Of the predictor we hold constant and make multiple plots Within each of those values. So this will construct the triptych. So in this case we start Here in the here we are in the at the top of the slide in the upper left We set water the centered version of it at minus one So that's the first level of water treatment the lowest water treatment We construct the predictions the counterfactual predictions across the three levels of shade And I've superimposed the data there the blue points For only for the plants that were grown at that water treatment Right, so you can see the ones that apply in that case And then I've just plotted this is the mu and it's and it's 95 percent Confidence, yeah Do you know how to do this in the code? There's nothing new. It's just the purpose that's new Then in the middle we set water to zero we redo the whole thing you can use the same code, right? And now I've only plotted the raw data For those plants that were grown at the middle water treatment where water is centered version of water is even zero It would be make sense And then once I get done with the model explain what's going on here. This is the non interaction model By the way, you're starting to catch on there in a bit So and then up here I set centered water to one the highest water level Only plot those plants grown at that level Across the three values of shade what I want you to see this is the non interaction model that I've done This trick the core so far notice that the slope is the same in every every panel. Yeah That's the same angle. Why because that's what a non interaction model assumes that the slope must be the same Regardless of the value of the other predictors The intercept is changing. So it's picking up something important as you add Water it keeps going up Right, and that's true. If you add water to two loops, they get their blooms get bigger and that is happening here But the effect of shade is assumed to be the same in every case And you can probably guess it isn't right just look at the data In that channel. It's not fitting very well. You can see that's a hint of the interaction Now let's plot the interaction model And I can show you what happens same code different model, right? At the lowest water treatment level cross values of shade Basically shade does nothing why they're known of water For light to help It basically does nothing at all these these plants are struggling to live And you can water them as much as you want, but that isn't what they eat, right? This is giving the starving man water in a sense And or or the thirsty man food. Yeah Sorry, yeah thirsty man food thirsty plant food and at the middle water level Across as the shade now we start to see the impact of denying them light They got enough water that they could viably produce some really nice flowers And then the ones grown in shade are performing worse. It hurts them, right? Because these are getting enough light to do something now because they've got some water to work with And oh, I wanted to say about this notice. It's probably not linear. This isn't a lot of data, but I I suspect there's nothing about tulips that forces them to obey linear relationships between the needs for growth, right? Especially you're zero it can't be linear, right? There's you can't get smaller than a zero blue and this model will happily predict a negative blue No problem And then at the highest water level shade makes the most difference when there's the most water You can do the most damage and it can do the most help But overall these are bigger, right because there's more water But the difference between the lowest shade level which means the most light And the highest shade level the least light is biggest here because the slope is That's the effect of the interaction Does that make sense? It's easy to appreciate this with a plot like this The coefficient table does nothing for you, right? There's like minus 50. Okay The minus 50 effective shade something great, but this tells you on the outcome scale What's going on and this is what we want and this is something your colleagues who Aren't as deep into the navel gazing of your study system as you are They can understand, right? Yeah question You could the question was what if you had more of a gradient of continuous values of predictors same thing There's nothing the discreteness here doesn't matter. There's it's still ordered So it'll work the same way if you have more possible values along the horizontal Works the same You may want to pick I mean what you pick on the top you have to make some decisions about the funds to put there so it might be median minimum and maximum or Medium and you know like the lower 10% and the upper 90% or something You have to make some decisions about where the bottom bounds are And you can explore a bunch of different ones And then you decide which ones to present based upon how they help your colleagues understand how the model is Does that make some sense? Yeah, yeah, exactly you could set the the sort of the unplotted predictor you could set it to the mean Whatever the mean is and then you could set it to you know the five percent quantile Of the distribution of that predictor two right, so it'll have a minimum Whatever the continuous range is to have a minimum you could put the minimum on the left here The median and or the mean in the middle and the maximum on the right the minute max I hesitate because they're often really extreme and unformative You could try something else I've already done this in a sense when I showed you with the ruggedness example and I flipped it and I showed you The effect of being in Africa depends upon ruggedness I already did this because ruggedness is continuous and there were a bunch of values and I had to choose some And I choose I think I forget what I chose. I think it was the 10 9 percent quantiles of it But it depends upon your domain knowledge of the system. I don't think there's the math doesn't tell us which values to show It really doesn't but you can't plot that would be that Well, you can with various tricks so like in in in this case It's easy because discrete in that ruggedness example. I did do this by fading out I use the alpha what's called the alpha transparency of the colors and I There's actually a function in the rethinking package that I wrote to do this because I'm a nerd But if you have a problem like that Give me an email and I'll we can play around with visual presentation and figure something out But I like the idea you fade out Cases in the data that are far from The value on the top of the plot Well, there's no sort of right answer depends upon the nature of the data Yeah Right right question was it doesn't matter which you choose we're going to do that next We're going to flip it and do burden's ass again. Yeah, exactly. Right. Do you remember the burden donkey? Maybe I don't know That was on tuesday where you're not here. Oh, okay. Sorry. No. All right. So, yeah, sorry You look shocked like who is burden and why are we talking about this ass? That's the thing about this this greeks greek logical paradox, which is no paradox at all No paradox is actually But uh So there's there's an ass then and it's hungry and it's echo this between two Piles of hay and so it starts death, right because it can't decide which direction to walk and interactions linear interactions Have no algebraic preference for which of these things you decide is conditional on the other because they're symmetrically conditional That was what we talked about on tuesday. So that was the callback joke to burden Okay, we're going to do that now. Uh, oh wait. No first thing we're going to do is I want to show you Remind you and you knew this but sintering doesn't change the predictions It seems and I want to emphasize this because it seems naughty, right? The data and you've transformed it by changing the mean this this something's wrong, right? It seems like cheating. There's this anxiety about it. It doesn't do anything Absolutely nothing neither to standardize Those are those are non-destructive transformations that don't change any of the information in the data Why because measurement scales are human inventions that don't really exist Right Out of myself as an anthropologist, right? So many of you beer sometimes at all, you know, we'll get into quantum physics really fast but uh No, I think measurement scales are frustrating because They they kind of get in the way of emphasis and we need them to collect data But then the actual operation of the system in reality that is outside of the constructions in our heads We used to represent it can't depend upon the measurement scale because we invented that it's just it's like a linguistic construction so Any kind of anything that just rescales is perfectly kosher and in fact there's a whole there's a very successful branch of analysis Mainly in the physical sciences called dimensional analysis or dimensionless analysis where you transform your model So that you get all the units out all the measurement scales go away And that is often the best way to analyze physical systems because again, you want to get the human invention out of the model of the system And again, if I had five more weeks on a semester system, I would do a day on that for you guys because there's some cool stuff You can do that way, but um Anyway, there's a pretty what I feel like you're taking like the log of something so shifting the mean does shift Oh, yeah logs Logs mean it's a different Hypothetical relationship when you take the log of a predictor put that in instead That that that means it's a different hypothetical relationship between the predictor and the outcome. Yeah, a multiplicative one Yeah, actually and often that's way more sensible in the business you and I do right the think that there's like a multiplicative relationship between the money In some outcome rather than an additive one, right? That's the thing that screw really thing, right? There's another hand. Yeah Using a measurement scale where zero has some important meaning Maybe if you don't have any The question was why did you have a measurement scale where zero has some intuitive meaning? If you want to use it go ahead That's fine too. I mean the flip side of it is Measurement scales are fine because there are inventions And choose an invention that helps you understand what's going on. There's nothing wrong with that Just saying there's nothing wrong with using another one too as long as it doesn't change the information Yeah, so it does seem a little weird, but anyway, so this is to show you Um Centering gives you all the same stuff. I'm just comparing on the top the uncensored models with the the triptych produce And then the center ones at the bottom all that happens is the exact values for the predictors have changed It's the same lines You do have to be careful sometimes if if your priors take into account the measurement scale And then you center or standardize your predictors. You can get different estimates if you don't also Rescale your priors. This happened to somebody in here with a homework problem At some point a couple weeks ago actually and it was a confusing at first and then I realized Oh, you resailed your predictors and your prior is still the same, but it should have different units too And that's what happened, right? So you didn't think that into account your prior has units So you have to keep that in mind if your priors are really flat like they are here You're not going to notice But if they're informative priors, uh, that just means you have to you have to pay attention to what's going on Okay Yeah, some animation. All right, they're the same All right, this is where we flip it and do the burden thing. Okay, so on the top We do it the other way now. We're going to take water and put that on the horizontal And put shade across the top and again, I think it's the same story, but this may be a different telling of it that Reveals something new to you about how it works And in this example, I like it because it's really symmetrical, right? They go in opposite directions So now we consider the question water depends upon shade and this triptych tells that story At low shade levels, meaning there's a lot of light increasing water helps a lot right, so At middle shade levels increasing water helps less And at high shade levels, increasing water does almost nothing Almost nothing at all. So the question is how does water depend upon shade? Water helps when there's a lot of light when there's not very much shade and then it has a big impact All right, can you do that right? That's why I use this example to understand the biology It's the same story as the bottom where shade depends upon water all the same cases all the same data are displayed But in a different relationship Now shade depends upon water so that when we increase the amount of shade At low water levels it makes no difference shade doesn't hurt Anything this says it slightly helps although you would try to get excited about that, right? And Oh now at middle water levels shade hurts a little bit And it definitely hurts a lot at high water levels. So you can see it both ways And it may be that you want to understand both of these things when you do an analysis of the system It depends upon what your questions are what you're focusing on so you can play around when I do these models I always compulsively do it all So I can make sure I understand what's going on and part of that is just checking that I've done it All right, I'm part of a model checking to see that look for stuff that looks like my computer malfunction Right And which happens to all of us This is make sense questions about this. Yeah, okay All right, uh, so Interactions are not always linear the ones we've looked at are Why are they linear because they represent a linear model of the coefficient that defines The the association between a predictor and the outcome. I'll say that again They're linear because we use a linear model to define the coefficient That defines the association between a predictor and an outcome as we take the ordinary linear regression model There used to be a beta coefficient there which was a parameter We replace it with a model and that model is linear and it's a linear function of the other predictor variable Right, and that's why it's a linear interaction And these things are geocentric, right? They're they're really powerful descriptive engines They're super useful unreasonably useful Given how goofy they are on the mid side, right? But I think there are lots of cases where you can do better by thinking a little bit about the biology Or the social dynamics of the system you work in And so let me give you an example in context of this We could add another kind of treatment variable to all these greenhouse experiments Assume all the ones we've looked at so far. We're grown at the ordinary cool temperatures that tulips like And and tulips are a high latitude flower. They're kind of winter rain flowering sort of thing in high latitude So that's why there's a runny and coin with a tulip on it That's where it comes from and that's what they're associated with lots of high high altitude Mid latitude blooming. So if it's hot they don't bloom This is this is why they grow them in places like, you know, lump of california and stuff like this where it's cool all the time and So you can imagine a greenhouse treatment where we really overheat the greenhouse And none of them bloom and all the bloom measurements are zero And you don't have to get tulips too hot on full before this is true Now they'll come up and they'll just sit there forever waiting for the winter rain Which never comes for sad things And uh, someone actually saw it No, it was great. Thank you Oh try it No, I love plants too. You're like deal for them, right? It's like I'm on grove That was great, so all right Yeah, so this is definitely an interaction It's an interaction because the effect of water and shade depend upon the temperature Right, if it's too hot neither of them matter at all and influence the bloom size It's just zero. It's flat lines. They have no effect Try to construct that as a linear interaction And in a moment I'm going to show you how to do three-way interactions So you'll see how you could do a potential three-way dependency among water shade and temperature But I assert here this is not linear because what you want is a step function of some kind where if the temperature gets high enough The other variables don't matter at all and it's just outcome is zero And that's what you want to predict and there are different ways to make models like that So if you find yourself in a situation where you've got something like this and and your knowledge of the modeling conflicts with your knowledge of the biology Give in to the biology and come talk to someone like me And that's what I'm going to do. I love projects like this working on phd students And there's something about your system and you know the biology and you don't like linear regression Those are the problems that are fun and You'll learn some modeling and I'll learn some biology and and everything will be great But there are different options. So I'm not going to suggest exactly how to do this But there'll be different ways to do it But linear interaction It'll probably pick up the dependency, but it'll make terrible predictions, right? Because it'll predict a continuous linear effect Of temperature change, which is not what happens in plants Okay, does this make sense? Yeah Okay, I wanted to show you three way interactions You can go as high ordering interactions as you like there's nothing special about Uh about two predictive variables interacting So remember with the simple interactions we've seen so far two predictive variables They address the question How does the influence how does the association between one of these and the outcome depend upon the value of the other? And symmetrically right same time Nothing wrong with three way interactions or even four way five way In theory you can go as high as possible in practice. You can't go very high Your computer will complain, but um And you'll lose your sanity and that's one of the things I want to show you here So how do you do it? This is a three way interaction model Let me let me show you the anatomy of it a little bit So the first part to realize is we've just got three main effects So we've got three predictors creatively called x1 x2 and x3 here. So this is like water shading temperature Each has got their their ordinary coefficients in front of them Then there are three two way interactions Each of these addresses The the dependency of one on the value of the other right and there are three combinations of the of the three Predictor variables right and then there's a term For the three way interaction itself which captures the three way dependency So how would you even think about what that means? I'll have some examples in a moment in the most abstract case We could be unintelligible, but I have to tell you before I do the the intelligible versions It would be that the influence of x1 or the association between x1 and the outcome y depends upon the interaction between x2 and x3 Joy right and what is the interaction x2 that is that the association between x2 and the outcome depends upon the value of x3 Okay, but so it's like layers deep of interaction and this is why these are hard to understand you can plot these and still get it Biologically, what is this capturing? It's capturing the sort of thing like in the tool of heat example if One of these variables changes and it changes The the interaction between the other two it makes the interaction between the other two predictors depend upon the value of the third predictor So the the interaction between shade and water, which means how to shade depend upon water, right? How's the effect of shade depend upon water that can depend upon temperature So it's perfectly credible and your systems may have things like this But they're they're pretty challenging to understand in in the model form and but plot it with plotting you can do it If you think you have to do this In they're very easy to specify In code though, so just showing you if you're going to do it with math you'd write it all out, right? Because I mean like that that's sort of how it goes, but With the automated formula tools in r you just multiply them together and this automatically implies all of the lower order Terms in this it'll this is just a three-way one But it'll do it'll populate all the possible two ways and the main effects as default when you do this And that's usually what you want, although not always later in the course in a couple weeks, I think We're going to have an example where I take out a main effect in an interaction model for theoretical reasons that we know It's zero And we're going to just remove it a priori But that'll make sense when we get there This makes some sense, you know for a moment. Let me give you a system Well, I'm going to give you the warning about this first and then I'll give you an example to think about a little bit. So higher order interactions Are pretty risky for inference. They're hard to think about they're easy to fit Now you can get kind of drunk if you've got a big data set You can just put in like a big four-way interaction and it'll be significant Because you've got a lot of data and then you'll report it and you'll feel like a superhero You'll have no idea what's going on or how to explain it two-way interactions are hard enough to talk through Right. I've stumbled over my explanations over and over again and I've talked this course a few times now So, uh, but it's still hard to talk it through through your interaction is even more confusing So, you know, the interpretation issue is is tough A surreal interaction says as I say here it it measures the extent to which the effect of x1 depends upon the value of x2 depends upon the value of x3 Right I assume people get this joke, right? This is a good movie. It's a quality movie And the bathtub scene especially is the best so hard to estimate These are hard to estimate. You need a lot of data with higher order interactions because it's cutting up the data Effectively is a smaller smaller and little bits which are relevant to any particular parameter That means you're getting big wide Posteriors intervals on those particular higher order interaction parameters. So there's not a lot of data relevant to calculate That's that's the big one information limitation. Also, there's this problem with multicollinearity, which we talked about before Interaction terms are necessarily correlated with their main effects. I'll say that again Interaction terms are necessarily correlated with our main effects. Why because they contain the same data Right, that's necessarily so so as you keep multiplying more and more predictors together Those correlations can escalate and be approach one shockingly quickly Especially for dummy variables So zero one indicator variables say you have three of them And this happens a lot in in political science. So you've got to code like gender race and income bracket And then you code in as dummy variables and discrete values those things interact massively in the American electorate Absolutely, and if you don't pay attention to those interactions, you can't predict how people vote Which is mainly on political scientists do as far as I can tell And I know some are listening. So I'm just like taking on them controlling them a little bit here, but But I think mainly like 90 percent of them do do that, which is fun. Someone's got to so As you multiply The only values are zero and one if you start multiplying strings of those things together They quickly approach being all zero Why because it only takes one zero to make them zero forever and then you can never get back out of there zero is a vortex, right and Then there there's no variation in the higher order interaction That you're basically throwing in a vector that's got like 90 percent zeros in a few ones There's almost no information in there and it's highly correlated with the predictors that went into it in the first place And often these things Generate they have that left leg right leg problem, right? It's redundant information That's put in you can make some progress here by regularization. That's the best thing Which you want to do you can also Standardize dummy variables. There's nothing wrong with that. I know it sounds creepy, but same thing applies It's an arbitrary measurement scale take your zero and values attract the mean to divide by the center deviation Or if we cool and it reduces the correlations because now there's just as many values on the negative side as the positive side and then you start multiplying negatives together and They flip sign and all kinds of good stuff stamps down the correlations. It's like algebra magic But it's nothing wrong with it. It changes the geometry of the thing in a really useful way. So Anyway, this is all to scare you which is what I mean to do here But if you really need these sometimes you really do so just think about it if you do Conditionality runs really deep in science causal systems If you push them in their extreme in any particular dimension the meaning of the other things change And that's just how nature works and that's part of the fun of studying it So if you're working in one of those boundaries where you've got A system where it gets pushed there sometimes And you need to do this then then you need to do it and then it's fine And if you're scared about it, you can ask people like me to help you with it Ideally your study takes it into account and gets an amount of data you need to make it work Power analysis is a great thing to do prospectively But but be aware that they're tricky. Okay. Let me give you last thing to say about interactions is Let me give you a thought experiment about through interactions. It's a data set It's built in to the rethinking package. You're not going to work with it on the homework You're going to go home and take a look at which is already up on the website But you are going to do it next week. You're going to work with this next week. I love this data set This is data from what's called the judgment of princeton which happened in new jersey in 2012 and it's like the judge famous judgment of paris where they uh Tested california wines against french wines, right and a california wines one and then suddenly california wines were all over the world market So they had one of these in 2012 because there are a bunch of new jersey wineries that are sure their wines are excellent and Everybody else has that reaction. He has the laughing exactly and uh So but they so they stage this and this is blind testing you get famous um Wine aficionados and known wine judges and they sit down and they sample the flights and they rake them numerically All the data is in this wines 2012 data set Including the names the unreactive names of the judges which i mentioned because After the results came out the french judges wanted their names redacted Because they liked new jersey wine So they were never going to live that down. I think that's the best part of this story Anyway, yes, french judges prefer new jersey reds to french reds This is a shocking thing. Anyway, not by very much. By the way, you'll see the data You'll analyze it in a couple weeks So the outcome variable here is score the rank numerical ranking they gave the wine and we have as predictors three binary ones and they all matter and and their own interactions in these data that arise from them There's region that the wine was grown in whether it was new jersey or france The nationality of the judge Right, you can might be worried that that matters That's usa or france in belgium Or there was a belgium judge which will count as france. It's like france with better french fries I'm trolling a belgium friend right now And flight red or white which were separated in scores. They didn't be separately for obvious reasons So if you want to think through the meaning of these in the context of your system You can verbally figure out what these interactions mean. So let's run through that exercise and this is going to reinforce that When you know about your system about its biology or the nature of the data It's easier to make sense of this and part of the the trouble with this introductory statistics courses is they're taught often Free of the context of your particular case and so you've got to teach them in the abstract and that makes it Hard it's the horoscopic problem as I keep saying so let's focus on these wines for a moment and I'll assume that most of you know enough about beverages that you can understand what's going on here I assume you all drink wine like beer because you're in california, but The predictors are region nationally of judge and flight we think it's a three-way interaction and Burden's ass still applies right now the donkey's got three piles of hay And it can't tell the difference between them and it starts to death So let's take them each in case and see the way you can interpret these interactions First there's an interaction between region and judge and we might call that the bias, right? The region why it's running and the identity of the judge those things the extent to which The region affects score depends upon the judge. That's because judges are fights Right. They like different kinds of wine. New Jersey Judges like New Jersey wine for example That interaction may depend upon the flight whether it's red or white Right. So the nature of bias depends upon flight Makes some sense. I don't know if it does when you'll analyze the data Interaction of judge and flight we might call preference Right judge some judges really like reds. Some really like whites and this is true in the data There are some judges who just like rate everything well, right? And That preference may depend upon the region the wine was grown in Right makes sense. So it's like this interaction depends upon the other thing You assign that interaction a name that helps you understand the context of the system Then you can reduce it down to another binary thing to help you think it through The interaction of region and flight I struggled to come up with any way to describe that and I just settled on comparative advantage doesn't sound smart Let's say some regions like New Jersey are better at growing reds with anything. It's probably true here and So you might say New Jersey has a comparative advantage in red lines But that advantage depends upon the judge because not all judges see it the same way like the same outcomes Makes sense So this is the kind of thing you can do when you know your system And you can go through and try to parse out this and when you explain your results to others This is really important, right? You get the summaries and say, okay, I drew all those plots So but what do we need? What's the take on message? Well take on message is New some regions have comparative advantages, but that comparative advantage depends upon the judge's identity Right because they don't all see it the same way. They have different preferences. Yeah makes sense Okay We're basically right on time shockingly Okay interaction is everywhere and as we move forward in the course starting next week We're going to have kind of a hiatus for new modeling types There'll be no new modeling concept, but we're going to get a new conditioning in a new way that's in our model So a new way to draw samples from posterior distributions called Marco chain Monte Carlo And that'll let us deal with these more complicated modeling types where interaction is always present Whether you want it to be there or not or whether you recognize it or not generalized linear models is what we'll do the week after next and In generalized linear models all the predictors interact to some extent even if you don't explicitly put any interactions You will still at various times, but and the reason is because The outcomes are bound to generalize on your models in some way. They're not continuous Equal density spaces like in Gaussian models. So think about an outcome like survival It's a zero on outcome that gets measured You're modeling the probability that something survives If some predictor is at a value such as the thing is almost certainly going to die It doesn't matter that you add some additional threat Right dead is dead. You can't be very dead It's like going on some cartoon like killed him a lot right with some cartoon saying So it's these things happen. You can't kill it a lot. It's either dead or not And that creates an interaction right the the effect of poison depends upon whether it's dead or not Right one easy way to think about it. So there's no avoiding the case in glms the interactions arise And we all have ways to cope with that. So don't worry. It's okay And it it reveals as as I like to say the majesty of nature in observing these crazy things because nature is full of of Feedbacks of this sort so And then generalize linear multi-level models Our one way to think about them is their massive interaction engines They let you take every parameter in the model and let it depend upon the identity of the unit that kind of produced it Whether that's an individual or a pond or a location A species Whatever it is you let the parameters the effects the treatment effects The average tendencies you let them vary by those micro details in the data There's a lot of power that arises from that we get better efficiency and estimation It's the main thing i'm going to push when we finally get there But you also get the ability to study variation, which is incredibly important Especially in observational systems where we can't squash the variation experimentally We just have to embrace it and measure it and try to describe as consequences Okay I think i'm i've got seven minutes, but I think i'll step a little early and just display king markoff And when you return on tuesday, I will pick him right here with king markoff And we will visit his island kingdom and learn How to do markoff king money parlor. All right. Thank you all