 regression and the content of today's lecture is basically hypothesis testing in multiple linear regression. We will be talking about how to test significance of regression model and test on individual regression coefficient and test for several parameter being zero using the extra sum of squares method. This is extra sum of squares method is a very important technique in regression analysis. Anyway, in the last lecture we have learnt how to fit a multiple linear regression model. So, given a set of observations on response variable and several regression variable, we know how to estimate the regression coefficients using least square method. So, once you have a fitted model, the next important job is to test the significance of the model. So, by testing the significance of multiple linear regression model, I mean whether there is linear relationship between the response variable and the regression variables. So, it might be the case that all the regressor given a problem, you know you are given a problem, you have one response variable and you have several regressor variables. So, it might be the case that all the regressor variables are irrelevant for the response variable. What I want to mean is that none of the regressor variables are significant for response variable. In other word, I can say that none of the regressor variables have contribution to explain the variability in the response variable. So, basically first we will be talking about test for significance of regression model. So, as I told that you know testing the significance of regression model, it means whether there is linear relationship between the response and any of the regressor variables. So, this is to test whether if there is linear between the response and any of the regressor variables x 1, x 2, x k minus 1. Well, so this can be tested by testing the hypothesis that H naught, which is the null hypothesis by testing this null hypothesis beta 1 equal to beta 2 equal to beta k minus 1 equal to 0 against the alternative hypothesis that beta j is not equal to 0 for at least 1 j. So, the null hypothesis says that you know there is no linear relationship between the response variable and any of the regressor variable and the alternative hypothesis says that no there is at least one regressor variable, which contributes significantly to the model. So, this is the hypothesis we want to test. So, the rejecting the null hypothesis here implies that at least one of the regressor variables x 1, x 2, x k minus 1 contributes significantly to the model. So, here b j is not equal to 0 means the regressor variable x j contributes significantly to the model. Well, so to test this hypothesis we will be taking the ANOVA approach. We know that the total sum of square is equal to s s regression plus s s residual. Well, now we also know that the s s t has degree of freedom n minus 1 s s residual has degree of freedom n minus k s s regression has degree of freedom k minus 1 s s regression by sigma square. This follows chi square distribution with the degree of freedom k minus 1 and also we know that s s residual by sigma square. This follows chi square n minus k and they are independent. Then from the definition of f statistics by the definition f statistics f is equal to s s regression by k minus 1 by s s residual. By n minus k this random variable follows f distribution with the degree of freedom k minus 1 n minus right and also it can be of course, we know that you know m s residual which is basically m s residual m s residual which is which is s s residual by n minus k. This is an unbiased estimator of sigma square. So, we know that expected value of m s residual by n minus k this is an unbiased estimator of sigma square. So, we know that expected value of m s residual is equal to sigma square and also it can be proved that the expected value of this one this one is nothing but m s regression expected value of m s mean square regression. This is equal to sigma square plus beta star dashed x c prime x c beta star by k minus 1 sigma square. So, well just I need to define what is this beta star and beta and x c basically b star is equal to beta 1 beta 2. I mean the beta vector was like beta naught beta 1 beta 2 beta k minus 1. So, beta star is obtained by excluding beta naught from beta and x c star sorry x c is equal to it has also been obtained from the x matrix, but this one is x 1 1 minus x bar that is the mean of the observations I mean all the regression variables and x 1 k minus 1 minus x bar. And x n 1 minus x bar x n k minus 1 minus x bar right. So, now look at these two expected value and these two expected values indicates that if the observed value of f is equal to is large because see f is nothing but this f is nothing but f is equal to m s regression by m s residual and here is the expected value of m s regression and here is the expected value of m s residual. So, if the observed value of f is large then f is equal to f there is at least 1 beta which is not equal to j. So, the higher value of f indicates that at least 1 beta j is not equal to 0. If you know see if all the regression coefficients beta j's are equal to 0 then this quantity is going to be 0 and f is going to be equal to 1. So, higher value of this observed value I mean higher values higher value of observed f indicates that at least 1 beta j is not equal to 0. So, if you know if you know based on this we reject h naught that is the null hypothesis that says that beta 1 equal to beta 2 equal to beta k minus 1 equal to 0. We reject this null hypothesis if f value is high means I mean f value is greater than f alpha and it has the degree of freedom k minus 1 n minus k. So, this value you can get from the statistical table well. So, now we just summarize the whole thing using the ANOVA table. So, here is the ANOVA table for multiple linear regression source of variations degree of freedom sum of square mean square and the f value well. So, sources of variation it could be the variation due to the variation that is explained by the regression the variation that is remain unexplained that is SS residual and this is the total variation total variation well the degree of freedom for this one is n minus 1 degree of freedom for regression is k minus 1 and residual is n minus k and this is called SS regression SS residual SS t. I have explained all these things you know what is SS t what is SS residual what is SS regression in the previous lecture. So, MS regression is equal to SS regression by the degree of freedom k minus 1 similarly MS residual is equal to SS residual by the degree of freedom n minus k. And here you have the f value which is equal to MS regression by MS residual and we know that this f follows f distribution with degree of freedom k minus 1 n minus k and we reject h naught if f is greater than I mean the observed f is greater than f tabulated f alpha k minus 1 n minus k. So, now next I move to the next lecture the test for test on individual regression coefficients once you determine that you know your null hypothesis in the previous test is rejected that means you know there is a linear relationship between the response variable and the regressor variable that means the null hypothesis is rejected means there is at least one regressor variable which has significant contribution to the response variable. So, once the null hypothesis in the previous test is rejected we know that there is at least one regressor which has significant contribution to explain the variability in the response variable well now the next obvious question is which regressor variable has significant contribution. So, we need to test know we need to test the regressor coefficients individually. So, test for test on individual regression and the visual regression coefficient. So, this is also it is called that it is called partial test partial or marginal test the previous one is called the global test I forgot to mention that well. So, here you know once you determine that you know at least one of the regressor variable is significant. So, the next question and which one is significant so we test for this one we test the hypothesis H naught which says that beta j equal to 0 against the alternative hypothesis H 1 that is beta j is 0. Not equal to 0. So, this basically you know this one and test this hypothesis test the significance x j in the presence of other regressor in the model. So, how to test this hypothesis that beta j equal to 0 against the alternative hypothesis that beta j is not equal to 0. We can we can go for you know of course, we know that the unbiased estimator we got is beta j which is equal to x prime x inverse x prime x inverse x prime x inverse x prime x inverse x prime y well and we know that this beta hat this follows normal distribution with mean beta and variance sigma square x prime x inverse that is what I proved in the last class. Now, from here we can conclude that you know beta hat beta j hat by sigma square. So, this is the total I mean this is the variance covariance matrix, but here we are only concerned about beta j. So, we will be taking the j j th element here. So, sigma square x prime x inverse you take the just j j th element. So, this one is nothing, but well. So, this follows we can say that this follows normal distribution with mean 0 and variance sigma square. So, now of course, this is sigma square is not known. So, if we if we replace this sigma square by m s residual then this is this variable or this random variable is going to follow t distribution well. So, the test statistic for this testing is that you know t equal to beta j hat m s residual x prime x inverse the j j th element of this follows t distribution with degree of freedom n minus k of course, this is under h naught. So, I did the mistake here you know this is this does not follow normal 0 1 this minus beta j this follows normal 0 1 and under h naught this beta j is equal to 0. So, under h naught you can say that this random variable follows normal 0 1 and of course, we are going to write this reject the non hypothesis. So, h 0 which says that beta j is equal to 0 we reject is rejected if the t value is far from 0 if this t value is greater than t alpha by 2 n minus k. So, this is the critical region to test beta j equal to 0. So, the first step is that you test whether the fitted model is significant that means, whether there is any linear relationship between the response variable and any of the regressor variable by using the global test that means, the hypothesis we tested at the beginning. So, if you see that yes there is a significant the test is significant that means, there is significant that means significant contribution of at least one regressor then you go for the partial test I mean once you know that at least one of the regressor variables has you know significant contribution to the response variable then you need to determine which regressor has the contribution and significant contribution to determine that you need to go for the partial test well. So, next we will move to we want to test for several parameters being 0. So, this is the technique here is called extra sum of squares method and this one has this very important application in a regression analysis. What we want to do here is that we want to test for several parameter being 0. So, what I want to mean by this is that well suppose you have multiple linear regression model say y equal to y equal to y equal to x beta plus epsilon here this beta is k cross 1 vector. So, it involves beta naught and k minus 1 regressor coefficients. Now, what you want is that we would like to determine if some subset of r of course, r is less than k minus 1 k minus 1 is the total number of regressors. So, if some subset of r regressors contribute significantly to the regression model. So, what I mean by this one is that suppose you know you have you are given a problem and in that problem there are 4 regressor variables and 1 response variable. So, you need to fit a model like y equal to beta naught plus beta 1 x 1 plus beta 2 x 2 plus beta 3 x 3 plus beta 4 x 4 plus epsilon and after fitting the model you know you feel that some of the some of the regressor variables are not significant may be you believe that x 4 and x 3 they are not significant. So, what you want to test is that whether beta 3 and beta 4 is equal to 0 against the alternative hypothesis that at least one of them is not equal to 0. So, you want to test the significance of beta 3 and beta 4 or x 3 and x 4 the regressor variable x 3 and x 4 in the presence of x 1 and x 2. So, extra sum of square technique is used to test such you know such hypothesis. Let me explain again suppose you have one response variable and you have 4 regressor variable x 1, x 2, x 3 and x 4 and you have to fit a model like beta equal y equal to beta naught and plus beta 1 x 1 plus beta 2 x 2 plus beta 3 x 3 plus beta 4 x 4 plus some epsilon. And you know after fitting the model of this form you know multiple linear regression model involving 4 regressors you believe you know you feel that x 3 and x 4 they are or for example, may be any subset x 1 and x 4 they are not significant. They do not have significant contribution to explain the variability in y. So, what you believe is that you believe that these two say x 3 and x 4 they are irrelevant for the response variable y. Well, so to test this one significant I mean to test this whatever you believe in statistically you need to test the hypothesis like h naught equal to beta 3 beta 4. You want to test that this vector is going to be equal to 0 against the alternative hypothesis that sorry beta 3 beta 4 against the alternative hypothesis that beta 3 beta 4 is not equal to 0 by not equal to 0 means at least one of them is not equal to 0. So, this is what you know we want to test this type of hypothesis can be can be tested using the extra sum of square method. Well, in general I have the model y call to x beta plus epsilon and my beta is a k cross 1 vector. Now, I want to split this beta into two parts I mean one is I will call it beta 1 the other one is beta 2. So, this is the beta 2 and this is the beta 1 is again a vector you know it is it has it is a k minus r cross 1 vector and beta 2 is a r cross 1 vector. And you believe that the last r reduced eraser variables are not significant for y using this partition. Similarly, we can we can divide the matrix x also x can be also partitioned into x 1 x 2 right. And then you can write this model as x 1 beta 1 plus x 2 beta 2 plus epsilon. And the hypothesis you want to test is that you think that this is enough you think that y call to x 1 beta 1 plus epsilon is enough for y. That means, what I want to mean by this one is that here in fact, I the analog thing is that I am testing beta 2 equal to 0. So, beta 2 is a vector which involves r regression coefficients. So, against the alternative hypothesis h 1 that y call to x 1 beta 1 plus x 2 beta 2 plus epsilon which is equal to 0. So, this in other word says that h 1 is that beta 2 is not equal to 0. So, what you claim is that this this we call this one you know the restricted model you feel that this restricted model or the first k minus r regressors are enough to the first k is equal to 0. Minus r regressors are enough to explain the variability in y you do not need the last r regressors well. And the alternative hypothesis says no the last r regressors are also significant to explain the variability in y. So, to test this type of hypothesis you know we use the extra sum of square technique. So, what we do here is that we compute s s regression for both the full and restricted well. So, this one is the full model this involves all the regressors and this is the restricted model which involve only k minus r minus 1 basically k minus r minus 1 regressors right well. So, one thing you have to understand that you know s s residual this one always decreases. As the number of regressor variable increases increases. So, this is a very intuitively of course, it is clear it says that the s s residual this thing decreases as you increase the number of regressor variables whether it is you know whether it is the newly added regressor variable is relevant for the response variable or not it does not matter. If you add one more suppose you have a model with k regressors. So, if you add one more regressor say if you make k plus one regressor variable in the model then s s residual decreases. But you know if the newly added regressor variable is significant or very relevant for the model for the response variable then it decreases more. But if it is not that much relevant for the response variable then the s s residual decreases less well the same thing you know since s s residual plus s s regression is s s total which is fixed. In other word I can say that s s regression increases as you increase the number of regressor variables. So, again the same statement the s s regression increases more if the newly added regressor variable is relevant to the response variable otherwise it increases less. So, this is the you know the basic idea behind the extra sum of square technique well let me compute let me compute s s regression for the full model first. So, s s regression full we know that the s s regression for the full model means by the full model I mean y equal to x beta plus epsilon. So, it has all the regressors. So, I know that the s s regression for the full model is beta hat prime x prime y minus n y bar square. You can refer the previous lecture for this one and this has degree of freedom k minus 1 right. Of course, you know y bar is nothing but 1 by n summation y i and also we know that m s residual for the full model is equal to y prime y which is this is basically s s residual y prime y beta hat prime x prime y by the degree of freedom because I wrote m s residual. So, the degree of freedom is n minus k right this is for the full model. Now, under h naught that is under the restricted model. So, restricted model says that y equal to beta 1 x 1 plus epsilon. So, we do not have the last r regressors in this model. So, under this restricted model my s s regression I said restricted. So, this is the notation s s regression under the restricted model. This is going to be beta 1 the same thing just I will replace beta 1 by beta 1 sorry beta hat by beta 1 hat and x by x 1 x 1 dashed y minus n minus n minus n minus n minus n minus y bar square right. And this has you know here you have you have not k minus 1 regressors you have k minus 1 minus r regressors in this model because we have removed r regressors from this model. So, this has this has degree of freedom k minus 1 minus n minus n minus r. Now, I said that s s regression increases as the number of regressor variable increases. So, that means the s s regression under the full model is greater than the s s regression under the restricted model because the full model has more regressor variable compared to the restricted model. So, this one you compute s s regression full minus s s regression restricted no this is called the extra sum of square due to beta 2 given that given that beta 1 is already model. So, let me this is very important let me explain little bit this I said that this is the extra sum of square due to beta 2 what is beta 2 beta 2 is the vector beta we have split it into two parts beta 1 and beta 2. So, beta 2 is the regressor beta 2 is a r cross 1 vector. So, it beta 2 is associated with the regression a regressor coefficient for those r regressor variables. So, this one is you know this is the s s regression for the full model s s regression for the restricted model. So, here you have all the regressor variable here you have the first k minus 1 minus r regressor variable. So, if you subtract this from here this becoming this will give you the extra regression sum of square I should say that extra sum of square is basically the extra regression sum of square due to due to the last k minus 1 minus r regressors. So, if well so I hope you understood. So, this is called the extra sum of square due to the last k minus 1 minus r regressors given that the first k minus the first sorry. So, this one is of order r cross 1 and this one is of order k minus 1 minus r regressors. So, this is the extra sum of square this is the extra sum of square due to due to the last r regressors given that the first k minus 1 minus r regressors are present in the model. Now, what we can do is that we can compute the degree of freedom of this one and the degree of freedom of this one is degree of freedom is see this has degree of freedom k minus 1 and this has degree of freedom k minus 1 minus r. So, this has degree of freedom r this minus this one. Now, s s regressors given that the first s s regression for the full model minus s s regression for the restricted model this is the extra sum of square and this has degree of freedom r this has degree of freedom r this by sigma square follows chi square r because this has degree of freedom r and s s residual for the full model by sigma square this follows chi square n minus k and you can check that they are independent. Now, we are in position to compute the f statistic which is equal to s s this extra sum of square s s regression for the full model minus s s regression for the restricted model you divide this quantity by the degree of freedom r by the definition of f statistics this follows chi square r. So, this random variable by r by this random variable by n minus k. So, s s residual full by n minus k you know this thing follows f distribution with degree of freedom r n minus k under. So, intuitively it is very clear that. So, this is see the numerator this portion is the extra regression sum of square due to due to the last r regressor variable. And if this quantity is more that means the last r regressors they have to have to have to have to have significant contribution in s s regression. That means they have significant regression they have significant contribution to explain the variability in y. So, intuitively it is very clear that if this quantity is large then we are going to reject the null hypothesis. So, the null hypothesis says that the null hypothesis says that you go for the restricted model, but alternative hypothesis says that you go for the full model well. So, if this quantity is large then if f is greater than f alpha r. n minus k then we reject h naught and conclude that at least one of the regressors in beta 2 is significant and conclude that that at least one of the regressor in beta 2 is significant. So, I hope you understood extra sum of square this is very interesting and also very important well that is all for today. Thank you very much.