 The last time we saw the Hoffman-Willand theorem which said that if A and E are matrices such that both A and A plus E are normal matrices and if lambda 1 through lambda n are eigenvalues of A and lambda hat 1 through lambda hat n are eigenvalues of A plus E then there is a permutation of the eigenvalues of A plus E such that each eigenvalue of A plus E is close enough to the corresponding eigenvalue of A in the sense that the sum of the squares of these differences is at most the spectral norm squared of E. So today we start on another topic which is the singular value decomposition. So far we have focused our attention on square matrices. So now we start discussing about rectangular matrices. Suppose so basically these are matrices of size m by n and m need not be equal to n. So one can view this as a linear map from the c to the n space to the c power m space. That is it maps an n dimensional vector to another m dimensional vector in the complex space. We have a few sort of preliminary or precursor lemmas and then we will go to the main theorem. Now there is one thing I want to mention which is actually not here. So remember that I mean we will be working with these with the induced spectral norm that is specifically A2 which is the max over norm x2 equals 1 norm of Ax L2. Now we know that induced norms satisfy some multiplicativity. Now when it comes to, so this is for square matrices. We had defined this for square matrices but we can always define something like this for rectangular matrices also. Okay because after all you can compute the Euclidean norm of any vector and Ax is a vector. The only thing is that this constraint space is over the space c to the n or x belongs to c to the n whereas the objective is being evaluated on a vector that is sitting in c to the m. So it is perfectly okay to define the norm A2 to be a quantity like this even if the matrix A is rectangular. Now we know that these induced norms satisfy the sub multiplicativity property and that extends also to rectangular matrices. So specifically this kind of norm it satisfies the property that AB L2 is less than or equal to A2 times B2. Of course here A and B are matrices that can be multiplied together. Okay so for example this could be m by n and this could be n by k or something. Okay then this is still true where these norms are evaluated as given above. Okay so this is one property and this is easy to show it's proof lies is exactly the same as the proof of the sub we showed a result that said that induced norms satisfy the sub multiplicativity property and the proof of this is exactly the same. Okay now a few other precursor lemmas that we need are that so if A is an m by n matrix then the eigenvalues of A Hermitian A are always non-negative. We know this already but anyway the proof is just one line if A Hermitian A times V equals lambda times V simply pre-multiply by V Hermitian you get V Hermitian A Hermitian AV equals lambda times V Hermitian V but lambda is just a scalar and this is just the L2 norm squared of AV and this is the L2 norm squared of V and both are therefore real and non-negative and since V is an eigenvector it's a non-zero vector so this is actually strictly positive. So this means that lambda see this is real this is also real value and so lambda cannot suddenly become complex value so lambda is and both are non-negative so lambda is real valued and it's non-negative. So now we formally define the singular values of a matrix so take a rectangular matrix A and the singular values which I will abbreviate as S vales of this matrix A denoted by sigma 1 through sigma n where these are ordered in that sigma 1 is the largest singular value and sigma n is the smallest singular value so I'm indexing it by the number of columns here okay and and sigma 1 squared through sigma n squared are the eigenvalues of A Hermitian A. So this is the crucial point here that sigma 1 squared up to sigma n squared these are the eigenvalues of A Hermitian A and we already saw that these eigenvalues are non-negative so I can take the positive square root and that gives me what the singular values are so it completely defines the singular values. Now another another lemma is that if A is of size m by n then rank of A is the same as rank of A Hermitian A which is the same as rank of A A Hermitian so A Hermitian A so if A is m by n A Hermitian A is of size n cross n and A A Hermitian is of size m cross m so rank of A is at most min of mn and it's also equal to the rank of this n cross n matrix and it's also equal to the rank of this n cross n matrix. I won't show this it's actually very easy to prove first of all rank of a matrix rank of AB is at most the rank of A and the rank of B so from that you get an inequality that says rank of A is rank of A A Hermitian A is less than or equal to rank of A and then to show the reverse inequality then what you do is you start with A Hermitian AB equals 0 and if that is true then V Hermitian times A Hermitian AB equals 0 or in other words AB that V Hermitian A Hermitian AB is nothing but norm of AB squared and if that is equal to 0 it means AB must be equal to 0 so the null space of A Hermitian A is therefore contained in the null space of A then you use the rank nullity theorem so you should write this out for yourself but it's not a difficult result to show and I have a few remarks if the rank of A is equal to R as a consequence of this if rank of A is R then the first R singular values of A will be positive and all the other singular values will be equal to 0 so the rank of the matrix determines how many nonzero singular values you will have and if A is a square matrix it's of size n cross n then lambda is a nonzero eigenvalue of A Hermitian A if and only if lambda is a nonzero eigenvalue of A A Hermitian as well and finally if A and A Hermitian they have the same rank and they also have the same set of nonzero singular values if m is not equal to n then one of them will have more zero suckering as singular values okay so these are some small remarks and you can actually relate to these after we go through the SVD theorem okay so here is the singular value decomposition theorem what I like about this theorem is that it applies to any rectangular matrix okay there's absolutely no structural assumptions being made so A can be any m by n matrix and let sigma 1 through sigma R be the nonzero singular values of A where R is equal to the rank of A and suppose these singular values are ordered so that sigma 1 is greater than or equal to etc greater than or equal to sigma R which is strictly greater than 0 because there are R nonzero singular values let D be a D cross D sorry R cross R diagonal matrix with sigma 1 through sigma R along the diagonal and zeros everywhere else and sigma is a matrix of size m by n with D as its stop left R cross R block and zeros everywhere else then you can find a unitary U of size m by m and a unitary V of size n by n such that U Hermitian AV equals sigma this diagonal matrix but it's not a square diagonal matrix it's a diagonal matrix of size m by n and it has sigma 1 through sigma R or as its stop left R cross R block and zeros everywhere else so that is what this theorem says so we'll theorem yeah so what is the difference between eigenvalue and singular value so the it you can see that here from the definition and we will discuss that a little more later but the singular values of a matrix squared are the eigenvalues of A Hermitian A okay for now this is the relationship that we've seen okay in general if since we are discussing rectangular matrices you can't talk about eigenvalues of A okay A x is not in the same space it's of size m whereas x is of size n so you cannot write an equation like A x equals lambda x is if A is of size m by n but if A is a square matrix we'll see what the relationship between the singular values and the eigenvalues of A are but for now we are discussing rectangular matrices and so we cannot write a direct relationship between I mean eigenvalues are not even defined for a rectangular A but when I take a matrix A Hermitian A that is a square matrix and I can define eigenvalues for it and the eigenvalues are all non-negative and if I denote them by sigma 1 squared up to sigma n squared then sigma 1 through sigma n will be the singular values of A that's the definition that's just directly the definition okay sir okay sir yeah go ahead please yeah can we say that the singular values are the positive roots of the eigenvalues of A Hermitian A that is right that is exactly what we are saying here that if I denote the eigenvalues of A Hermitian A which are all non-negative by sigma 1 squared through sigma n squared then sigma 1 is the positive root of those eigenvalues of A Hermitian A sigma 2 is the positive square root of the second largest eigenvalue of A Hermitian A and so on