 Okay, so we have already seen this beautiful factorization of any arbitrary m cross n complex matrix called the SVD which is to say that suppose you have matrix like so, we have seen that we can always write this matrix in the following form, okay, where u is of size what is the size of u? m cross m, right? v is also of size n cross n. So, these are square matrices and sigma, sigma 1, sigma 2 until sigma r and then you have zeros elsewhere. Of course, this is real, right and these are in fact the singular values of the matrix A. What is the size of this? What should be the size of this? m cross n, right? These are real numbers. Additionally, the important thing to note here is that sigma 1 is greater than or equal to sigma 2 until sigma r which is strictly greater than 0, right? What else? What can we say about this u? These are unitary matrices, right? Which is to say that further u Hermitian u is equal to u u Hermitian is equal to an identity of size m and v Hermitian v is equal to v Hermitian is equal to an identity of size n, right? So, we have seen this. Now, in view of this I shall now write this down in a slightly different form, okay, but one that you should be immediately able to relate to from this itself which is to say that A is equal to perhaps I should denote it as something like this. So, this is the summation not the sigma, alright? This is summation i going from 1 through r sigma r u r v r Hermitian or rather I think the other day I actually denoted these by y r's, right? Where of course u is equal to y 1, y 2 so on until y r and then y till day r plus 1 until y till day what was the size of u m and v is equal to v 1, v 2 until v n, clear on this? Why this is obvious? You see what is this? The way I would write this expression down here I can just stack up the column side by side and then stack up the rows of v with their conjugations one on top of the other like so, right? And then this form results. You might wonder what happens to the remaining I am taking a sum from, oh sorry I should just say i, right? Yeah, that is a mistake. Yeah, that is a running variable. So, I am only taking r. You might wonder what happens to the rest. Well, the rest if you look at this turn out to have no weight except 0. So, the columns of u past this y r the rth column because what is sigma doing when acting on u? So, if I just look at this much part of it then every time I pick a column of the sigma it is basically filtering out one column of u and scaling it by some factor. What is that factor? The factor is exactly the singular value that is the amplification of the gain that is being applied to that direction. So, directions beyond the rth direction which is this do not get amplified at all. They have gains of 0 is it not, right? So, therefore we can conclude that this sum will only run up to r and this is beautiful because what this says is that this matrix A can be written as the sum of r rank 1 matrices. Think about a typical object like so, y i v i Hermitian. What does it look like? So, this is y i 1 y i 2 until what is the size of y? It is an m tuple, right? y i m and this is nothing but v i 1 conjugate v i 2 conjugate times v i n conjugate. So, the resultant is a matrix whose first row is just y i 1 times this particular row. The second row is y i 2 times this row. So, every row is just a scalar multiple of another row that means there is just one row which is linearly independent all other rows are linear combination in this case precisely a scaled version of that one row. So, this is a rank 1 matrix, yeah? So, the rank of each such object is obviously equal to 1. So, in other words what this SVD has allowed you to do is to write any matrix A, yeah? As a sum of r rank 1 matrices. So, rank of this matrix this is called an outer product by the way that is besides the point anyway. So, the first row of this resultant is just y i 1 times this row, second row is y i 2 times this row. So, every row is a scaled version of this row. So, there is just one linearly independent row the row is span the entire row span is span by just this row alone. So, therefore, the rank is equal to the dimension of the row span which is equal to that you can also look at the column picture. The first column is just v i 1 conjugate times this the second column is just v i 2 times this. So, the entire column span the image is span by just this, right? So, each of these individual elements sitting inside the sum is just sigma i times this and sigma i is not equal to 0 for i running from 1 through r. Therefore, this is a sum of sum of r rank 1 matrices and now go back to the question we had yesterday in the previous day where we said what is the best rank k approximation. So, out of these rank r is it not the case that you pick out the ones with the highest weights, right? You remember that rank k approximation of m cross n matrix that is what you did you can just write it down as this, right? And then pick out the largest the directions corresponding to the largest singular values sigma 1 through sigma k and just truncate it there, right? Now, why it works again we will not prove that as I said in the previous day, but that is indeed a very useful application of the SVD which allows you to compress the amount of data contained in an m cross n matrix through just m plus n times r number of data points, right? So, as promised in the previous day we shall now prove the fact that and that this form is going to be very handy in proving that that the kernel of A or rather maybe I will start with the image first. So, the image of A is nothing but the span of y 1, y 2 till y r. So, that is basically the first r columns of u will give me what exactly the image of A. Now, why is that so? So, again as we have done so many times previously when we try to prove this equality of subspaces we have to show containment both sides, right? So, consider A V yeah any object of this form A V this definitely belongs to image of A, alright? So, this is nothing but summation sigma i y i V i Hermitian times V i going from 1 through r. Now, this V comes from where it is an n tuple what is the basis for C n that is pertinent to us? We have a basis for C n just like the columns of u are a basis for C m. The columns of V do they not also by our very construction in the previous day have we not seen that the columns of V provide us with a ready made orthogonal basis or orthonormal in this case orthonormal set of vectors that form a basis for C n, right? So, using that result I am going to write this as summation sigma i y i and then summation V i Hermitian alpha j V j. So, I am just going to push this V i Hermitian inside I could have kept it outside also, but you see because the key is to get these V's to interact with each other where these V j's are what? The columns of V, okay? Columns of V is this clear? So, this essentially together with the summation is my V. Any vector V that I pick out in C n the n tuple of complex numbers can be represented as a linear combination of the columns of V because V is an orthogonal matrix. Therefore, it forms its column span not just span not just the arbitrary span they are actually given orthogonal basis for the vector space that is C n, right? In that case what happens unless i is equal to j because of the orthogonality it is always going to be 0, right? Please ask if this is not clear. So, this is where I am saying j going from 1 through n, right? j is the second running variable the summation on j essentially captures V. So, maybe I should write V is equal to summation alpha j V j, j going from 1 through n, yeah? Clear so far? Please ask if there is any doubts, alright? Now, what do you think will be left behind because this V i is remember V i is fixed. So, when I am operating with this V i all the alpha j's apart from alpha i have vanished is it not? So, each of these objects corresponding to an i only filters out one object from these V j's which is the if object. So, next this sum becomes irrelevant because it is not going to be a sum after all after you carry out this inner product this is an inner product the conventional inner product on C n. So, after you carry out this inner product you are not going to be left to the sum anymore, but you just going to be left behind with alpha i and V i Hermitian times V i is just unity because these are orthonormal basis. So, this entire thing just devolves to alpha i, right? So, this is going to be summation alpha i sigma i y i i going from 1 through r. Of course, then it stands to reason that this belongs to the span of or rather here I have used the other notation for span. So, this obviously belongs to the span of y 1 y 2 through until y r. So, one way containment is shown that the image is contained inside the span of this, right? No doubts on this so far. Any questions? This is actually also a very interesting thing here you see especially if you are talking about signal processing and things like that and if this corresponds to different channels different directions. So, if alpha i happens to be your gain along the ith direction. So, your input could be some vector of signals and alpha i is the gain along the ith direction then that is amplified by sigma i. So, your output is along y i for an input along V i your output is along y i. So, it exactly tells you that this V i is rotated in a certain way of course, the spaces have changed. So, your input is probably an n tuple and your output is an m tuple. So, you cannot talk about it as a pure rotation in the same vector space, but if it is square of course, you can talk about it. So, basically along the ith direction whatever input you give with an amplification with an amplitude of alpha i that gets amplified by sigma i and it is along the ith output direction which is given by the vector u. So, there is a very nice connection between these channels a channel input along V i reflects in an output along y i with an amplification sigma i. So, in a MIMO system whether it is signal processing whether it is in control theory multivariable controls wherever you come across these things if you freeze the instant of time that at every fixed instant of time you have this sort of a thing happening. So, this captures that phenomena very nicely, but anyway our concern is not that our concern was to show this equality and so far we have managed to show one sided containment. So, we have shown that the image of a is contained inside the span of y 1 y 2 through till y r right. Now it is time for the other part of the story. So, what we now have to show is does there exist V such that for all alpha 1 alpha 2 until ok. So, we are now going to cook up spanning set for the y's right alpha r we have a solution to the equation which equation exactly that A V is equal to summation alpha i y i because if so then does not matter what vector I pick out in the image or rather what a vector I pick out in the span sorry what vector I pick out in the span of these y i's I can always represent it as the image of some vector under the transformation A right. So, if the answer to this question is a yes if the answer to this question is a yes we would have proved the other side of the containment you agree that is exactly how to look at it, but I would not you know I would just use this step here. So, any A V if it has to satisfy that condition what I am asking for is this to equal this is it not. So, I am saying summation i going from 1 through r sigma i y i V i Hermitian V is equal to summation which is equivalent to this summation alpha i y i you see what I am going to do now it is fairly obvious I mean I just require one solution I do not care about a unique solution you see and in fact there would not be unique solutions anything that maps anything that gets mapped to this you add something in the kernel of A that will also map to this. So, the kernel of A is non trivial then there will be multiple solutions, but I am just happy with just any one solution right. So, let us just say that I choose choose yeah. So, what is my choice I have in my choice this V I am required to choose a particular V. So, choose V such that the inner product of V with V i is equal to alpha i by sigma i can I do that well of course, I can because I am only looking for i is equal to 1 2 until r for which I know that the sigma i's are all non zero. If I choose V in this manner what happens I mean it is just a trick the what would satisfy my deal was obvious from this step itself see what is what is happening the projection of V along these principal directions is tailored in such a manner that what will happen yeah. So, this V i Hermitian V is nothing, but this inner product no. So, it is designed by by by very design it is meant to satisfy this equality I can of course, keep adding something more to this V as a linear combination of V R I mean yeah as in terms of the remaining terms in the vector in the matrix V the remaining columns of V beyond the first R columns, but it would not matter right. In fact, in our next part of the proof we shall show that those columns of V will exactly be the kernel the basis for the kernel of A. So, they would not matter right. So, this choice such a V satisfies let us call this equation as star. So, such a V satisfies star. So, I was required to find a solution V such that for any given sigma summation of alpha i y i I would be able to satisfy the condition that A times V is equal to that. So, anytime I pick out a vector in the span of the y i's the first R y i's the first R columns of U I should be able to find some V that satisfies that condition. And now here I have at least one such solution no claims for uniqueness, but at least this solution serves my purpose yeah you will see that just plug it in just understand that this is nothing, but this inner product just plug it in here alpha i y sigma i. So, the sigma i and sigma i will cancel on we have the alpha i and that is it right. The fact that the sigma i is a non zero for the first R values one through R just works all the way for me. So, in fact you have the other sided containment as well y 1 y 2 until y R. So, anytime I pluck out a fellow from this span it is also going to be part of the image of A. So, what is the dimension of the image of A then. So, the big statement is any rank R matrix can be written as a sum of R rank 1 matrices by our observation. This is a sum of R rank 1 matrices and the representation there of A through this SVD is basically that of a rank R matrix. In fact, the number of non zero singular values is exactly the rank of a matrix equivalently right when you are dealing with complex or real fields it is the beauty of the singular value decomposition that allows you to deal with most properties of matrices in a very I should say elegant fashion many of those properties are very apparently obvious right. So, now we will try and see how to characterize the kernel of A. So, by rank nullity what should be the dimension of the kernel of A yes. So, it is an remember m cross n matrix. So, the ambient space from which it is mapping is n dimensional right it is plucking out vectors in its domain which are n n tuples of complex numbers. So, it is n dimensional over the complex field yeah. So, n is equal to R plus nullity or the dimension of the kernel. So, the dimension of the kernel is n minus R. So, the first thing about A is that the dimension of kernel A is equal to n minus R clear on this is from R and T rank nullity theorem straight away clear no doubts about this why this is n minus R right because our V is now n dimensional V is C n right ok. Now, let us consider V which is given by V 1 V 2 until say V R and then V R plus 1 till V n remember these are orthogonal set of vectors right. So, any time I pick out consider A V k for k greater than R what can I immediately say A V k is equal to summation sigma i y i V i i going from 1 through R times V k right. This is nothing but summation sigma i y i times V i inner product with V k i is equal to 1 through R. What are these inner products remember i is going from 1 through R, but k is something greater than R. So, due to the orthogonality I would posit that this is 0. So, each other individual term in this summation is 0 yeah therefore, this is 0. So, this implies V k belongs to kernel A whenever you take k greater than R that V corresponding to that k V k must belong to the kernel of A which means which essentially means that we now have V k plus 1 V k plus 2 until V n the span thereof is contained in kernel A right. Each of them individually belongs to the kernel. So, you take their linear combinations which is what a span is all about that will also belong to the kernel, but what is the dimension of this what is the dimension of this. Isn't that a set of linearly independent vectors I mean they are orthogonal after all. So, any set of orthogonal vectors unless there is 0 there is no 0 vector over there. So, obviously it is linearly independent. So, this span has n minus R. So, this should be R right yeah sorry it should be R yeah probably that is what you are pointing out yeah thank you. So, this must be R. So, this must be equal to n minus R and the dimension of the kernel of A is also n minus R. So, we do not need to show both show both sided inclusion here all that we need to show is that this object is sitting inside this and the object sitting inside has the same dimension as the extension thereof. Hence, we must have kernel of A is equal to the span of V R plus 1 V R plus 2 all the way until V n right. So, basically the first R objects in the column span on the among the columns the first R columns of U provide you a basis for the image of A and the last n minus R columns of V provide you with a basis for the kernel of A yeah and not just any arbitrary basis, but these are orthogonal basis. So, it is beautiful the way the singular value decomposition sort of characterizes any linear transformation it gives everything in a very crisp and transparent fashion, but of course you have to assume that you are dealing with complex or real matrices only right because that is when this question arises these inner products and these orthogonalities and all of this only makes sense right. So, with that we will bring the brief journey of course there is lots more I mean singular value decomposition is probably one of the most useful factorizations you will learn about matrices, but so obviously it is there is lots more that you can delve into and we can talk about this for hours on end, but we will have to bring it to a close here because we have been a little way lead from our main topic which was the study of Eigen values and Eigen vectors. So, presently we shall move into that domain. So, here is where we shall put an end to our brief journey into singular value decompositions ok. So, there and back again right.