 Hi, we are studying numerical methods for computing eigenvalues and eigenvectors of a given matrix. In this we have introduced power method in the last class and also we learnt convergence theorem for power method. In this class we will study a small variant of power method called inverse power method. Let us first recall a result from linear algebra which states that if lambda is an eigenvalue of an invertible matrix A, then 1 by lambda is an eigenvalue of A inverse. This is not very difficult for us to see you just take lambda as the eigenvalue of a matrix A, then you can write A into V is equal to lambda V, where V is an eigenvector corresponding to the eigenvalue lambda. Since A is invertible, you can write V is equal to lambda times A inverse into V. That implies A inverse into V is equal to 1 by lambda times V. Here you can note that there is no change in the eigenvector, only the eigenvalue becomes the reciprocal of the eigenvalue of the matrix A. So, that is very important for us to note. Using this theorem, we can in fact compute the smallest eigenvalue of a matrix A. When I say smallest eigenvalue, it should be smallest in the absolute value. Say for instance suppose we have a matrix A whose eigenvalues are rearranged such that modulus of lambda 1 is greater than or equal to modulus of lambda 2 and so on up to lambda n where mod lambda n is strictly less than mod lambda n minus 1. Now you can see that if A has such a eigenvalues, then lambda n which is the smallest eigenvalue is unique. That is the only eigenvalue which is smallest in this sense. Now let us see how to compute this eigenvalue lambda n using power method for that what we will do is instead of applying power method to A which will capture the dominant eigenvalue. Now we will apply the power method to A inverse. Why we do that? Well we have seen from the above theorem that if lambda n is the smallest eigenvalue of A, then 1 by lambda n will be the dominant eigenvalue of A inverse. That makes us to apply power method on A inverse in order to compute 1 by lambda n and therefore, we can obtain this smallest eigenvalue of A. That is the idea. Remember once we compute 1 by lambda n, the power method applied on A inverse will also capture an eigenvector corresponding to 1 by lambda n. That is the same as the eigenvector corresponding to lambda n also. So, in that way we can obtain the smallest eigenvalue of A and a corresponding eigenvector of the eigenvalue lambda n. So, that is the interesting part of power method and this way of computing the smallest eigenvalue of a matrix is what we call as the inverse power method. Now let us give the iterative procedure for inverse power method. You can see that there is nothing much difference in inverse power method when compared to power method. The only difference is instead of applying the power method to the matrix A, now we will be applying it to A inverse. That is for every iteration we will compute y k plus 1 as A inverse x k. Remember in power method we will define y k plus 1 is equal to A into x k right, but here we will apply A inverse to get y k plus 1. Well this is not something which is straight forward for us because we are given the matrix A right. Now in order to compute y k plus 1 we need to invert the matrix A which may be computationally costly. So, what we can do is you can write this equation as A into y k plus 1 equal to x k. Here you can observe that once you choose x naught you will plug in x naught on the right hand side which is a known vector and you will then solve this linear system to obtain the unknown y k plus 1. That is a major difference in inverse power method when compared to the power method. This complicated step is not there in power method because y k plus 1 is just defined as A into x k. x k is known to us and A is also given to us therefore, y k plus 1 can be explicitly computed from there whereas, in inverse power method y k plus 1 has to be computed by solving this linear system. That is we have to solve this linear system y 1. Once you have this then you will have x 1 and that has to be then taken as the right hand side to get y 2 and so on right. So, if you see here the right hand side vectors will only change whereas, the coefficient matrix A will remain unchanged. Therefore, what you can do is you go for a LU factorization because once you do the LU factorization A is equal to L into U then the system A y is equal to x can be solved by first doing a forward substitution L z is equal to x and that gives you z then we can plug in that into U y is equal to z. So, you bring that to the right hand side and that gives us the required vector y. We have learnt it in one of our previous classes where we have discussed LU factorization you just recall that. So, the advantage of going for LU factorization is that once you make the factorization L into U for the matrix A then at every iteration you just have to plug in the right hand side vector and do one forward substitution and one backward substitution you will get your y that will be computationally relatively cheaper. So, once you get this y k plus 1 rest of the algorithm reminds the same as we did in the power method. What you have to do you take this vector y and find the maximum norm of that and in whichever index the maximum norm is achieved you take that value of the y coordinate and define that as mu k plus 1. For instance if y k plus 1 is equal to say minus 2, 3 and minus 5 then your capital J which is nothing, but the minimum of all the j's at which the maximum norm is achieved. So, here it is achieved only at the third coordinate therefore, mu k plus 1 will be y 3 k plus 1 in this particular example it is minus 5. So, this is just an example. So, in that way only you have to choose mu k plus 1 once you have mu k plus 1 well you can find the x k plus 1 very easily by taking y k plus 1 and divided by mu k plus 1 which will be a unit vector. So, we have seen all this when we were discussing power method and this is the iterative procedure for the inverse power method. The only difference and the complication is in computing y k plus 1 otherwise all these steps reminds the same as in the power method. Well the next question is if these sequences that is sequence mu k and sequence x k if both these sequences converge then where do they converge? That is the question. In a sense we have already answered this question, but we will make it more precise. Of course, we have to keep in mind that the initial guess x naught should be chosen in such a way that C 1 is not equal to 0. What is C 1? C 1 is nothing but the coefficient in the representation of x naught in terms of the eigenvectors C 1 v 1 plus dash dash dash up to C n v n in that the coefficient C 1 should be non-zero that is for the theoretical reason, but assume that everything went well with our choice of x naught. Then our question is where these two sequences will converge? Well the sequence mu k will converge to the dominant eigenvalue of A inverse which is nothing but the reciprocal of the smallest eigenvalue of A. In our rearranged form it is lambda n therefore, mu k will converge to 1 by lambda n. Then where does the sequence x k converge? x k will converge to an eigenvector of lambda n. Remember it will converge to an eigenvector of 1 by lambda n which is of course, the same as the eigenvector of lambda n also. Let us see an example. Let us take the matrix A given like this. I am just giving you only 4 digits after the decimal place. Well as I told you we prefer to first decompose the matrix A into L into U. I have just taken the do little factorization. You can just observe it here all the diagonal elements are 1. So, I have just taken the do little factorization for A. So, once you have L and U then A into y equal to x can be done with a forward substitution and one backward substitution and at each iteration you can get the vector y. Once you get the vector y then you will look for that coordinate of y at which the maximum norm is achieved. If it is achieved at multiple coordinates then you will take the one which is with the minimum coordinate. So, in that sense the maximum norm is achieved at the first coordinate of y 1 and therefore, mu 1 is given by minus 2.14. Remember it should be minus naught plus and once you get that then simply take x 1 as y 1 divided by mu 1. So, that is your x 1 and that is given by this and once you have x 1 again you plug in x 1 into L into U into y 1 is equal to x 1 right. A forward substitution and the backward substitution will give you y 2 sorry this is y 2 right. So, you will get y 2 from y 2 again you will find the coordinate at which the maximum norm is achieved that is mu 2 and then you divide mu 2 y 2 by mu 2 you will get x 2. Now, you repeat this process I am just going head with the iteration you can observe that mu is converging slowly to the number 1.63. Remember if mu is converging to 1.63 it means what it is actually converging to 1 by lambda 3 right. What is lambda 3 well we have not given what are all the Eigen values of A therefore, we do not know that is the smallest Eigen value of A right. When I say smallest it is the smallest value of all the Eigen values in the absolute sense right. Therefore, this is for us given as 1.63 when you take the reciprocal of that that gives you 0.61 and what is the corresponding Eigen vector well for Eigen vector you do not need to do anything just comes as it is the Eigen vector is taken as minus 0.81 and 0.03. So, this is the smallest Eigen value of the matrix A and a corresponding Eigen vector of this Eigen value is given like this and this is how the inverse power method will go on well for this we have chosen the initial guess as x naught is equal to 1 1 1 I am sorry I forgot to mention this. The next question is ok we learnt how to capture the dominant Eigen value of a matrix and the smallest Eigen value of a matrix. Now is there a way to find some of the other Eigen values of A also that is the question well this can be done by shifting the matrix A appropriately and that results into shifted inverse power method. Let us try to understand what is shifted inverse power method assume that we are interested in capturing an Eigen value lambda j and one of its corresponding Eigen vector say v j. Now what you do is you choose a nu some real number nu such that mod lambda j minus nu is strictly less than mod lambda i minus nu where i ranges from 1 to n with i naught equal to j. It means you try to find a nu such that when you take this quantity that will be the smallest value of all such quantity when applied to all other Eigen values of the matrix A well practically this is not possible to get because simply we do not know what is lambda j. However in certain situations we may use Grashgorian disc to get an estimate of nu. For instance suppose your matrix A is such that the Grashgorian disc or something like this disc 1, disc 2 and so on, disc 3 and then disc 4 like that, disc 5 like that and then in between you have a small disc which is disjoint from here and you want to find this Eigen value let us call this as our x lambda j. In such cases you know how to choose your nu say you take this interval and suppose this interval is very small then you have an idea of how to choose this nu. So, like this in certain restricted situations you may be able to choose such nu using Grashgorian theorem if such a favorable situation is not happening with the matrix A then you can also try with A transpose sometimes that may give you a better information about how the Eigen values are distributed. So, if such a favorable situation happens then one may go to use shifted inverse power method otherwise it is not that easy for us to use this method in any practical situation ok. Assume that we got this nu somehow then how to compute lambda j is our question now. Once you have such a nu then probably by looking at this you can get an idea of how we are going to capture lambda well what you can do is you apply the inverse power method to A minus nu i. So, this is called the shifted matrix and you apply inverse power method to this matrix inverse power method is applied to A minus nu i that is equivalent to saying that apply power method to A minus nu i inverse right. That is the idea we do not need to give the procedure for this because it is just the same as what we have given in the inverse power method only thing is instead of A inverse now we have to put A minus nu i inverse that is the only difference. Well let us consider this example now once you have this you will have to use a appropriate nu as I told you this is generally not possible perhaps if you are very lucky you may get an idea of how to choose this nu from Grashgorand deals of either A or A transpose. Well I will leave it to you to think how to do that for some appropriate matrices we will give you some such matrices in exercise problems, but here to keep our discussion very simple let us know what are all the Eigen values and the corresponding Eigen vectors of this matrix A these are given by lambda 1 lambda 2 and lambda 3 and the corresponding Eigen vectors are given here. Now let us try to capture lambda 2 and Eigen vectors of lambda 2 let us see how this is going to happen for that we have to choose a nu right let me choose nu as minus 2.4 you can choose any number in such a way that lambda 2 minus nu is strictly less than lambda 1 minus nu and this is strictly less than lambda 3 minus nu also. So, this choice of nu is going to work therefore, I chose like this you can also choose any value and now we have to apply the inverse power method for A minus nu i that is A plus 2.4 i. You can clearly see that the Eigen values of A minus nu i are given like this. Now we have to find the LU factorization of A minus nu i I have gone for the du-little factorization you can go for either Gaussian elimination or Crude factorization or even if the matrix is symmetric and positive definite you can go for Cholsky's factorization also right, but I have gone here for du-little factorization as per this the lower triangular and the upper triangular matrices are given like this and the iterations are given like this. I have stopped this iteration procedure at the fifth iteration because you can see that from the fourth iteration to fifth iteration there was no much improvement at least to the number of digits that we have shown here that is why I have stopped up to here. If you want more accurate then you have to go for more iterations that is the only need here, but I am happy with this because it is just an illustration. So, I stop here therefore, your iteration mu k sequence is converging to remember it is going to converge to 1 by lambda 2 minus nu right. Remember we are now working with shifted matrix therefore, your mu k is going to converge to 1 by lambda 2 nu and since mu phi is given by minus 7.61 you can see that 1 by mu phi is something like of course, 1 minus 1 by 7.61 that is in my calculation approximately equal to minus 0.1314 right. So, that is going to be approximately equal to lambda 2 minus nu right because mu is going to converge to lambda 2 minus nu therefore, 1 by mu k will converge to lambda 2 minus nu I have just stopped at the fifth iteration therefore, I am going to consider this value as a approximate value to lambda 2 minus nu, but remember nu is taken as minus 2.4 right therefore, lambda 2 is equal to minus 0.1314 right plus nu that is nothing but minus 2.4 and that is equal to minus 2.5314. So, that is the lambda that we are trying to obtain from the shifted inverse power method. Let us see what is the value of lambda that we have taken previously that is also approximately the same it is minus 2.5313 what we got is minus 2.5314 right. So, we are pretty good in our approximation and the corresponding Eigen vector captured by our shifted inverse power method is 1 comma minus 0.06 comma 0.04 you can also see that it will be a scalar multiple of the corresponding Eigen vector that we have chosen in our first slide right. So, this gives us an idea of how we may capture other Eigen values of a given matrix, but there is a serious disadvantage in this approach that we do not know how to choose this scaling parameter nu. Apart from that the method is very interesting with this we will finish our discussion or power methods. I thank you for your attention.