 Today, we are going to start a new topic and that is the Eigen value problems. So, far we considered real vectors, real matrices. Now, even if matrix is a real matrix, its Eigen values they can be complex. So, that is why now our underlying field is going to be complex numbers. So, we will be considering complex matrices, then the vectors also will be complex. Eigen values they are defined for square matrices. We will show that if a is n by n matrix either real or complex, then its roots or the Eigen values they are given by roots of a polynomial of degree n. Now, as a consequence of fundamental theorem of algebra, we know that if a polynomial has degree n, then it has got exactly n 0s or n roots counted according to their multiplicity. That means we will count if a 0 is repeated twice, it will be considered as 2 0s. Now, when we consider polynomial of degree bigger than or equal to 5, then we cannot have a formula for finding its roots. Like if you have got a quadratic polynomial, then we can write its 2 0s in terms of the coefficients of our polynomial. If you have got a x square plus b x plus c is equal to 0, then the roots can be written in terms of the coefficients a b c. This will not be possible when your polynomial is of degree bigger than or equal to 5. So, that is why for calculating the Eigen values our methods, they are going to give us only approximation. This was not the case with solution of system of linear equations. When we considered Gauss elimination method or its variants, then the error came because of the finite precision. Whereas, the method was exact method in contrast for Eigen values our method will be giving only an approximation. So, one tries to find as much information possible as of Eigen values by say looking at a matrix. So, there are some special matrices for which we will study what are their Eigen values. That means, we can if the matrix is a real symmetric matrix, then its Eigen values they are going to be all real and similar results. Then, we will have some localization results. That means, we will find a region in the complex plane, which is going to contain all our Eigen values. We are going to consider power method for finding the Eigen values dominant Eigen value of a matrix and then there are some variants of this method. I am going to describe what is known as QR method for finding Eigen values. At present, that is the most popular and the best possible method available for calculating Eigen values or rather calculating approximations to Eigen values of our matrix A. Now, it is beyond this course to prove convergence of QR method. The description of QR method can be given easily and that is what I will do. So, now, we are going to start with complex vectors when we consider the real vectors and complex vectors. For real vectors, what we had done was you can add two vectors. So, that is component wise addition. You multiply a vector by a scalar. So, you multiply each component of your vector by that number. So, these things remain same for complex vectors. It will be the real numbers they are replaced by complex numbers. So, again addition of two vectors will be component wise. Multiplication by a scalar will be same as before. Then, matrix into vector multiplication will be exactly same as before. There will be a change in the definition of inner product because we have to take into consideration the complex numbers. Then, we had defined one norm, infinity norm for real vectors that definition remains exactly the same. The corresponding induced matrix norm, the proof will have slight modifications, but let me not get into those details. If they are the formula which you obtain is exactly the same as before. So, now, let us quickly consider complex vectors. Then, the inner product, the vector norm and the matrix norm. So, let us look at the complex vectors and the corresponding operations. So, we have got z to be a complex vector z 1 z 2 z n. So, each z i is going to be a complex number. W is another n by 1 vector. As I said before, z plus w will be component wise addition. So, it is z 1 plus w 1 z 2 plus w 2 plus z n plus w n. Alpha times z will be each component will get multiplied by alpha. Then, inner product. So, here when we had real vectors, then the inner product was x comma y was summation x i y i. Now, here change is you will consider z i w i bar. W i bar is the complex conjugate. Now, when you consider inner product of z with itself, it will be summation i goes from 1 to n z i z i bar. So, you have complex number, you are multiplying by complex conjugate. So, it will be summation i goes from 1 to n mod z i square. So, thus inner product of z with itself will be bigger than or equal to 0 and it will be equal to 0 if and only if z is a 0 vector. When you consider inner product of w with z, it will be summation w i z i bar by our definition, which will be same as summation i goes from 1 to n z i w i bar and then complex conjugate. So, that means it is z comma w bar. So, we have got conjugate symmetry inner product of w with z is complex conjugate of inner product of z with w. This is linearity in the first variable z plus v w will be summation i goes from 1 to n z i plus v i into w i bar split the summation into two summations. The first summation will be nothing but inner product of z with w and the second summation is inner product of v with w. Similarly, if you consider alpha z comma w, this will be summation i goes from 1 to n alpha z i w i bar. Now, alpha is independent of i. So, it will come out of the summation sign what remains in the summation that is inner product of z with w. So, our inner product will be linear in the first variable. So, these are the properties of the inner product. The first is positive definiteness, second is conjugate symmetry and the third property is linearity in the first variable. When you consider z comma alpha w, then alpha will come out as alpha bar because of the conjugate symmetry. So, inner product is conjugate linear in the second variable. So, this is the difference between real inner product and complex inner product that real inner product was symmetric. Now, this is conjugate symmetric and we had linearity in both the variables for real inner product. Whereas, now complex inner product is going to be linear in the first variable whereas, conjugate linear in the second variable. Otherwise, it is exactly similar. Now, we had Cauchy-Schwarzen equality for real inner product. So, there is Cauchy-Schwarzen equality for complex inner product also and using this Cauchy-Schwarzen equality, one considers the induced norm. So, that is induced norm by the inner product. So, one shows that it satisfies various properties of norm. So, here is inner product of z with z is summation i goes from 1 to n mod z i square. We define norm z 2 to be positive square root of z comma z and Cauchy-Schwarzen equality is modulus of z comma w is less than or equal to 2 norm of z into 2 norm of w. I want you to notice that our complex inner product, it is a map from c n cross c n to c. So, in general, our complex inner product is a complex number, but when you consider inner product of a vector z with itself, then it is going to be a positive real number and that is why you can take its positive square root and then obtain a real number. In fact, the real number is going to be bigger than or equal to 0 and that is our Euclidean norm. So, norm z 2 is positive square root summation i goes from 1 to n mod z i square. Norm z 2 will be bigger than or equal to 0. It will be equal to 0 if and only if z is equal to 0 vector that will follow from positive definiteness of inner product. Norm alpha z will be equal to mod alpha times norm z. So, it follows from the definition and the triangle inequality norm of z plus w is less than or equal to norm z plus norm w. So, it is for the triangle inequality that we need the Cauchy-Schwarzen equality. So, this is about the 2 norm. Now, analogously, one can define 1 norm and infinity norm. So, norm z 1 is equal to i is going to be summation i goes from 1 to n mod z i and norm z infinity to be maximum of modulus of z i 1 less than or equal to i less than or equal to n. So, in the definition there is no difference. Instead of real numbers, we have got complex numbers, but you are taking its modulus. For 2 norm, we are taking summation mod z i square. So, this modulus is important. For real inner product space or for if the vector is real, whether I write x i square or whether I write mod x i square, the answer is the same. Whereas, for the complex number, it is important that you should take modulus of z i square. Now, we are going to look at the induced matrix norm. So, if you are given any vector norm, you define norm of the matrix to be maximum of norm a x by norm x x naught equal to 0 and then for 1 norm and infinity norm. That means, if you are taking or if you are fixing vector norm to be 1 norm, then look at the corresponding induced matrix norm. For that, we obtained an expression in terms of the elements of the matrix. Similar thing was possible for norm a infinity, whereas for the 2 norm, we have to be satisfied only with an upper bound. So, here the expressions for norm a 1 and norm a infinity, they are going to remain to be exactly the same. So, we are looking at the induced matrix norm. So, we have norm a 1 to be column sum norm. So, summation i goes from 1 to n modulus of a i j. So, look at the first column, take the modulus, add it up, do it for all the columns, whatever is the maximum that is norm a 1. Norm a infinity, the expression is obtained by interchanging j and i. So, column sum norm becomes row sum norm. So, we have got norm a infinity to be summation j goes from 1 to n modulus of a i j, 1 less than or equal to i less than or equal to n and then this is the Frobenius norm. So, it is summation over i summation over j mod a i j square raised to half. Norm a 2 is not compatible, but norm a Frobenius, here it is norm a 2 less than or equal to norm a f, here this less than or equal to should be bigger than or equal to. Then we have got this basic inequality, norm a is maximum of norm a z by norm z. So, from here we get norm a z to be less than or equal to norm a into norm z for z belonging to C n. Next we define conjugate transpose. So, we define the conjugate transpose for a vector as well as for a matrix. So, you take complex conjugate of each entry and then you take transpose. So, if you are taking conjugate transpose of a vector, column vector, then its conjugate transpose will be a row vector. If the matrix is square matrix, then conjugate transpose is again going to be equal to the matrix of size n. So, this conjugate transpose, we know that matrix multiplication is not commutative. So, if the conjugate transpose commutes with the matrix, then it deserves a special name, it is a special class of matrices and those are known as normal matrices. So, we are going to define normal matrix and then self adjoint matrix, q self adjoint matrix. These matrices, their Eigen values, they have got some special property. So, here is definition, z is vector z 1 z 2 z n, z star is z bar transpose, z becomes a row vector z 1 bar z 2 bar z n bar. Now, inner product of z bar transpose becomes z with w, this is our definition summation z i w i bar. So, in this notation, we can write it as w star z, w star is going to be a 1 by n vector, z i is n by 1 vector. So, when you take 1 by n vector multiplied by n by 1 vector, you are going to get 1 by 1 matrix or you are going to get scalar. So, inner product of z with w will be same as w star z, next for a matrix a, we define a star to be equal to a bar transpose conjugate transpose. If you repeat the operation, a star star is going to give you back matrix a, then when you consider a b star, this will be a b bar and then transpose, a b bar will be same as a bar into b bar and then it is transpose. When you take a bar b bar transpose, the order gets reversed. So, you get b bar transpose, a bar transpose. So, this will be equal to b star a star. So, a b star is b star a star and inner product of a z with w, will be, we have seen that this is w star a z, then w star a, I write as a star w star because when you take the complex conjugate, it will become w star a star star, that means, w star a and this way is nothing but z comma a star w. So, important property a z comma w, a will go to the second variable as a star and here are the special matrices a star a is equal to a a star. So, that is class of normal matrices, then a star is equal to a that is class of self adjoint matrices. If you consider a star is equal to minus a that is skew self adjoint and lastly unitary matrix. So, we have got a star a is equal to identity and now for matrix we know that the left identity is same as the right identity, left inverse is same as the right inverse. So, that is why you will have if a star a is equal to identity, then automatically a a star is equal to identity. Now, if you take two self adjoint matrix, if you add it up, then again you are going to get a self adjoint matrix, the this result will not be true for product of matrices because when you will consider a b star, then you are you are going to have b star a star. So, if a star is equal to a b star is equal to b does not mean a b star is equal to a b because a b star will be equal to b a. So, these are some of the special matrices and they are going to their Eigen values, they are going to be something special or we can say something more about their Eigen values. So, now we want to show we want to define Eigen value Eigen vector and then we want to show that they are roots of a characteristic polynomial. So, here is Eigen value problem, our notation is going to be a will be either a real matrix or a complex matrix, but it has to be a square matrix. One defines Eigen value and Eigen vector only for square matrix. So, definition is a complex number lambda is said to be an Eigen value of a. If there exists a non-zero vector u such that a u is equal to lambda u and in that case u is called an associated Eigen vector. This non-zero part is important because if you take a zero vector, then when you apply a matrix A to it, you are going to get a zero vector, then a u will be equal to lambda u for any lambda. So, lambda will be Eigen value provided you have got a non-zero vector u such that a u is equal to lambda u. Now, how to find a lambda like you cannot find, but at least we want some characterization. So, that characterization we are going to show that the lambda is nothing but look at determinant of A minus lambda i. A is matrix which is given to us, then you look at matrix A minus lambda times identity. Look at it is the determinant is something which we can calculate. So, you will get a polynomial in lambda of degree n and our Eigen value is going to be zero of this polynomial. So, we start with the definition that lambda is Eigen value provided we have got a non-zero vector u such that a u is equal to lambda u. So, we have a u is equal to lambda u u not equal to zero. This will imply that A minus lambda i u is equal to zero vector which will mean that A minus lambda i it is a n by n matrix. So, we can consider it as a map from C n to C n. Any vector in C n you apply A minus lambda i to it you again get a n by 1 vector. So, A minus lambda i from C n to C n it is a map. This map is not one to one because we have got A minus lambda i u is equal to zero vector where u is a non-zero vector and A minus lambda i into zero vector is also equal to zero vector. So, we have got two vectors u bar and zero vector which have the same image and that is the zero vector. So, that is why A minus lambda i will not be one to one. If A minus lambda i is not one to one it cannot be invertible because for invertibility what we need is our map should be one to one and on two and in our case in finite dimensional spaces it is sufficient that if A minus lambda i is one to one then A minus lambda i will be invertible or if A minus lambda i is on two it will be invertible. So, our we are starting with lambda is an eigen value u is eigen vector. So, we have got A minus lambda i. So, map A minus lambda i will not be one to one. That means, A minus lambda i will not be invertible. So, you have got A minus lambda i to be a singular matrix. Now, if it is singular that means its determinant has to be equal to zero. So, you get determinant of A minus lambda i to be equal to zero. Now, conversely suppose lambda i is a complex number such that determinant of A minus lambda i is equal to zero. So, you look at homogeneous system A minus lambda i z is equal to zero vector. Now, this homogeneous system it is going to have a non-trivial solution because the coefficient matrix has determinant equal to zero. So, it has a non-trivial solution u such that A minus lambda i u is equal to zero vector and that precisely means A u is equal to lambda u u not equal to zero vector. So, thus the eigen values of A they are given by determinant of A minus lambda i is equal to zero. So, this is the determinant of A minus lambda i when you will expand the determinant you are going to have minus 1 raise to n lambda raise to n plus c n minus 1 lambda raise to n minus 1 plus c 1 lambda plus c 0 is equal to zero. So, you have a polynomial in lambda of exact degree n because the coefficient of lambda raise to n is non-zero it is minus 1 raise to n. Now, by consequence of the fundamental theorem of algebra this it is going to have n roots if you count them according to their multiplicity. So, thus we know that the n by n matrix it is going to have at the most n eigen values and they are going to be roots of this polynomial. So, thus the problem of finding eigen values it gets reduced to finding roots of a polynomial. So, this determinant of A minus lambda i this polynomial now we factorize it. So, it will be lambda 1 minus lambda raise to m 1 lambda 2 minus lambda raise to m 2 into lambda k minus lambda raise to m k where the m 1 m 2 m k they add up to n. So, you have got eigen values to be lambda 1 lambda 2 lambda k these are distinct eigen values and the power m i that is known as the algebraic multiplicity of lambda i. So, you count lambda 1 m 1 times lambda 2 m 2 times and lambda k m k times and that is how you have got exactly n eigen values counted according to their algebraic multiplicity. Now, there is another multiplicity associated with eigen value and that is known as geometric multiplicity. So, your geometric multiplicity is going to be number of linearly independent eigen vectors associated with a particular eigen value. So, we have a u is equal to lambda u u not equal to 0 vector if I consider a of alpha u this will be alpha times a u a u is lambda u. So, it is alpha times lambda u now alpha and lambda they are scalars those are complex numbers. So, they commute and then you can have lambda times alpha u. So, if u is an eigen vector alpha u will also be an eigen vector provided alpha is not equal to 0. So, eigen vector is not unique you have got infinitely many eigen vectors as soon as you find one eigen vector any non 0 multiple of it is also going to be an eigen vector. Now, one defines what is known as eigen space. So, see what you have got is suppose I have got a eigen vector then I take a multiple. So, if you are in say r 2 you are going to have a straight line except what you do not want is multiply by 0. So, eigen space by definition is going to be all multiples and you add 0 to it. So, all non 0 vectors in your eigen space they are going to be eigen vectors associated with eigen value lambda. So, there are infinitely many eigen vectors but when you consider number of linearly independent eigen vectors they are going to be finite and in fact that that number is going to be less than or equal to algebraic multiplicity. So, if you have got lambda 1 to be an eigen value with algebraic multiplicity to be m 1 in that case you can have at the most m 1 linearly independent eigen vectors it the number can be less. We will consider an example where your number of linearly independent eigen vectors can be strictly less than algebraic multiplicity. Your algebraic multiplicity is you consider factorization of characteristic polynomial and in that you have lambda 1 minus lambda term whatever its power that is our algebraic multiplicity and geometric multiplicity is number of linearly independent eigen vectors associated with it. So, here is definition of eigen space null and the sub space of a minus lambda i is set of all z such that a minus lambda i z is equal to 0 vector it is a subspace it consists of eigen vectors and a 0 vector the dimension of this subspace is called geometric multiplicity of our eigen value lambda then as I said it is same as number of linearly independent eigen vectors associated with eigen value lambda and geometric multiplicity will always be less than or equal to algebraic multiplicity. So, now let me give you an example of 2 by 2 matrix a simple matrix for which in one case geometric multiplicity is strictly less than algebraic multiplicity and in another case they are equal. If your matrix is upper triangular matrix then your eigen values are going to be diagonal entries. So, for upper triangular matrices you do not have to do any computation just look at the diagonal entries those are your eigen values. Now, when you considered Gauss elimination method we reduced matrix a to upper triangular form, but these elementary row transformations they do not preserve the eigen values you have matrix a it has got certain eigen values you do elementary row transformations obtain an upper triangular matrix, but the eigen values of upper triangular matrix which you have obtained will be completely different than your original eigen values. This elementary row transformations they do not change the solution of system A x is equal to b that is why it was useful there whereas, here it is not useful. So, now let us consider a example. So, here is upper triangular matrix 1 1 0 1 the determinant of a minus lambda is 1 minus lambda square. So, a has eigen value 1 with algebraic multiplicity 2. So, it is a repeated eigen value I look at it is eigen vector. So, 1 1 0 1 u 1 u 2 is equal to u 1 u 2. So, you get u 1 plus u 2 is equal to u 1 and u 2 is equal to u 2 this second equation gives us no information the first equation tells us that u 2 has to be 0 that means null space of a minus i vector u 1 0 u 1 belonging to c. So, your null space of a minus i which is all u 1 0 u 1 belonging to c. So, that means we have got multiples of vector 1 0 if you want eigen vector then it should be a non-zero multiple. So, for this example you have got 1 is eigen value with algebraic multiplicity 2 and geometric multiplicity to be 1. So, geometric multiplicity is strictly less than algebraic multiplicity. Now, let me change this example slightly let me make this 1 as 2. So, when you look at matrix 1 1 0 2 it is characteristic polynomial will be 1 minus lambda 2 minus lambda. So, you have got eigen values to be 1 and 2 with algebraic multiplicities in both the cases to be equal to 1. When we try to consider the eigen vector then you are going to have u 1 plus u 2 to be equal to u 1 and 2 u 2 is equal to u 2. So, that means u 2 has to be 0 and eigen vector will be of the form u 1 0 with u 1 not equal to 0. So, 1 will be eigen vector with geometric multiplicity to be equal to 1. Next, look at 1 1 0 2 u 1 u 2 into is equal to 2 times u 1 u 2. So, what will be the first equation? It will be u 1 plus u 2 is equal to 2 u 1. Second equation will be 2 u 2 is equal to 2 u 2. So, again the second equation does not give us any information from the first equation you will get u 1 is equal to u 2. So, any eigen vector associated with 2 will be of the form u 1 u 1 u 1 not equal to 0 or equivalently it is going to be a non-zero multiple of vector u 1 1. So, eigen vector of 1 will be 1 0 or any multiple eigen vector of 2 will be vector 1 1 and any non-zero multiple. So, now what we are going to do is we are going to consider eigen values of our special matrices. If the matrix is self adjoint a star is equal to a then we will show that eigen values they have to be real. If a star is equal to minus a then eigen values have to be purely imaginary or 0. For normal matrix we do not have any such structure. Your eigen values can be complex, but still for eigen values of normal matrix it has got some special property. If you look at two distinct eigen values and corresponding eigen vectors then they are linearly independent. For normal matrix something more is true eigen vectors corresponding to distinct eigen values they are going to be non-zero. So, they are going to be perpendicular to each other that means their inner product is going to be 0. If you consider eigen vectors of unitary matrix that means the matrix which satisfies a star a is equal to a star is equal to identity then the eigen values they are going to have modulus to be equal to 1. So, they will lie on unit circle. Now, what does this eigen values tell us? So, these are going to be precisely the points where a minus lambda i will not be invertible. At all other complex numbers our matrix a minus lambda i will be in the invertible. So, when you have got n by n matrix there are going to be at the most n complex numbers for which a minus lambda i will not be invertible. For all other complex numbers a minus lambda i will be invertible. So, let us show the properties of eigen values of special matrices. The proofs are simple and straight forward. So, look at a u is equal to lambda u u not equal to 0 vector lambda complex number pre multiply by u star. So, you have got u star a u is equal to u star lambda u. So, which is same as lambda times u star u. u star u will be summation i goes from 1 to n u i u i bar. So, that is summation i goes from 1 to n mod u i square u is not a 0 vector. So, that means at least 1 u i will be non 0 and hence this summation will not be equal to 0. So, I get lambda to be equal to u star a u divided by u star u which is equal to in the notation of inner product it is a u comma u divided by u comma u. So, we have lambda to be equal to inner product of a u with u divided by inner product of u with u. Let me consider complex conjugate of lambda this is going to be complex conjugate of a u with u divided by complex conjugate of u with u. Now, since inner product of u comma u is bigger than or equal to 0 here this u comma u bar will be same as u comma u. So, this is a u and by conjugate symmetry the numerator is going to be inner product of u with a u divided by u comma u. So, thus lambda is equal to a u comma u divided by u comma u and lambda is equal to a u comma u divided by u comma u and lambda bar is u comma a u divided by u comma u. Now, lambda is also equal to this a when it goes to the second variable it goes as a star. So, it is going to be u a star u upon u comma u. Now, from here I can conclude that a star is equal to a will imply that lambda bar is equal to lambda and which will mean that lambda is going to be real because lambda is a complex number it is complex conjugate is equal to itself that means lambda has to be real. Similarly, if a star is equal to minus a then your lambda bar is minus lambda. So, if lambda is equal to x plus y. So, it is say minus x plus y and lambda bar is going to be x minus i y and hence in this case you are going to have if you have got a star is equal to minus a then lambda bar is equal to minus lambda and this means that lambda is purely imaginary or 0. So, this is for self adjoint and skew self adjoint matrices now for the normal matrix. So, suppose a is normal. So, you have got a a star is equal to a star a. Consider norm a x its Euclidean norm and its square this will be nothing but inner product of a x with itself this a will go here as a a star. So, it is x a star a x now use the property that a star a is same as a a star. So, it will be x a a star x which will be x. Now, this a I can write as a star its star a star x. So, this is same as a star a star x a star x because this a star will go to the second variable as its star. So, this is nothing but norm a star x to norm square. So, an important relation that if a is normal then Euclidean norm of a x is same as Euclidean norm of a star x. How does this property helps us for saying something about Eigen values. So, what we have proved is if a is normal then norm a x is same as norm of a star x. Then suppose lambda is Eigen value of a then we have got a minus lambda i u is equal to 0. So, norm of a minus lambda i u will be equal to 0 now a normal will mean that if I consider a minus lambda i its star that into u also will be 0. So, that will mean that lambda bar will be an Eigen value of a star. So, a normal implies norm a x is equal to norm of a star x its two norm then a u is equal to lambda u u not equal to 0 vector. So, norm of a minus lambda i u will be 0 this will be same as a minus lambda i star u will be equal to 0 and this is equal to a star minus lambda bar i u is equal to 0 and thus a star u is equal to lambda bar u. So, now for normal matrices the a and a star if lambda is Eigen value of a lambda bar will be Eigen value of a star and Eigen vector is going to be the same. So, using this fact in our next lecture we will show that Eigen vectors of a normal matrix associated with distinct Eigen values they are perpendicular then I am going to state Schur's theorem spectral theorem and then we will go to localization of Eigen values. So, thank you.