 Okay, so the next topic is generalized inverses of matrices. So we know that if a matrix is square and non-singular, we can always invert it. But in order to solve systems of linear equations and even for many other problems, we will need to invert matrices that are rectangular of size m by n. So for that, we use this concept of generalized inverses and these generalized inverses have various properties that we can also study. So we will start with the basic definition. The generalized inverse of a matrix A of size m by n is the unique matrix B of size n by m. So note that the dimensions are reversed, it's of size n by m, satisfying four properties. And first property is that AB is a Hermitian symmetric matrix, that is AB equals AB Hermitian, which of course is B Hermitian A Hermitian. Similarly BA is a Hermitian matrix, that is BA is equal to BA Hermitian, which is equal to A Hermitian B Hermitian. Then A times B times A equals A, B times A times B equals B. So the matrix B that satisfies these four properties is a unique matrix and is called the generalized inverse or the Moore-Penrose pseudo inverse of this matrix A and we often denote it by a dagger. Okay, so the fact that this is a unique matrix that satisfies all these four properties is the essence of this proposition. So let AB a matrix of size m by n, then there exists a unique matrix B which satisfies one to four. So what it says, in fact saying two things, one is that it's always possible to satisfy these properties one to four and there's only one matrix that satisfies all four of these properties. Okay, very interesting proposition. So in other words, you can find a pseudo inverse for any matrix A, no exceptions. And that pseudo inverse that you find, there's no other one, it's a unique one. Okay, so here is how the proof goes. So again, we start with the singular value decomposition of the matrix A, U sigma V Hermitian. So first we'll talk about existence. That is to show that there exists a matrix B which satisfies these four properties. Sir. Yes. Excuse me. Go ahead. Sir, does it mean that any square matrices of non-full rank, there exists this pseudo inverse? Can you repeat your question? If A is a square matrix of rank, lesser than full rank, I mean not full rank, does pseudo inverse exist for that also? Always. Okay, thank you. Yeah. So you can find a pseudo inverse of any matrix, square, rectangle, anything, it always exists and it's unique. You can't find two of them. Okay. It's a very intriguing property. Okay. Sir, one more question. Yeah. Does the pseudo inverse equal to inverse? Yes. So we'll see that actually, that when, if the matrix A is square and invertible, the pseudo inverse equals the inverse. In fact, we can see something about that right away. Because A was a square matrix and it was invertible. And if B was equal to A inverse, then if I take AB, AB is the identity matrix. Yes. Of course, it is Hermitian. Similarly, BA is the identity matrix. Of course, it is Hermitian. If I do ABA, that BA cancel with each other and I'll be left with A. If I do BAB, AB cancel with each other, I'll be left with B. So the normal inverse also satisfies all these four properties when A is square and invertible. That is the reason why it's called a generalized inverse. Okay. It's not telling you something new when you go to square matrices that are invertible. It's the same inverse that you will get if you were to invert the matrix. So this sigma here has an R cross R diagonal block D in the top left where R is equal to the rank of A. So that is our standard SVD. Let sigma 1 be an N by M matrix, okay, with D inverse in the top left R cross R blocks and everywhere, zeros everywhere else, okay. So this is of size M by N, okay, and sigma 1 is of size N by M, so the opposite size. What we'll first show is that sigma 1 is the Moore-Penrose pseudo inverse of sigma. So if I take sigma 1, okay, that is equal to sigma 1. So this you just have to write it out what it looks like. So this will be like a D with 0, 0, 0 here and then this sigma 1 will be D inverse. These zeros are all of different dimensions. This is of size N by M. This is of size N by N. So I'll get the identity and zeros zeros and that is exactly equal to sigma 1 Hermitian times sigma Hermitian, which is to say that the Hermitian of this matrix is equal to itself, okay. So sigma sigma 1 is Hermitian, so it satisfies property 1, sigma 1 sigma is equal to sigma Hermitian, sigma 1 Hermitian. So this also satisfies that this matrix is a Hermitian symmetric matrix. Sigma times sigma 1 times sigma equals sigma. So I'll just write this like this. This is actually a bad way to write it because this sigma, this is sigma 1, this is sigma 1 Hermitian and this is sigma Hermitian. So they're all of different dimensions but you have so basically the zeros are all of different dimensions but by just matching up the dimensions you can verify that this is actually true. And similarly sigma sigma 1 times sigma is equal to sigma and sigma 1 sigma times sigma 1 equals sigma 1. So basically sigma 1 is the Moore-Pendreau's, okay. So now since I have it said shown uniqueness, I can say that it is Moore-Pendreau's inverse of sigma. Sir, is it the Pendreau's who got the Nobel Prize? Okay. So coming back to our proof, suppose B, now okay, so let's define B to be V times sigma 1 times U Hermitian, okay. So what is B? If I have the SVD of A, I'll write it here, A is equal to U sigma V Hermitian. All I have done is to take the non-zero r cross r block in sigma, invert that and put that as a matrix sigma 1 of size n by m, pre-multiplied by V and post-multiplied by U Hermitian, okay. Interesting matrix. It's easy to obtain once you have the singular value decomposition of A. Then we can easily check that B satisfies properties 1 through 4. That is, for example, if you look at AB, that is U sigma V Hermitian V times sigma 1 times U Hermitian just substituting for A and B. But V Hermitian V is the identity matrix. So it's sigma sigma 1 U Hermitian. And sigma sigma 1, sigma sigma 1 is equal to sigma 1 Hermitian sigma Hermitian. So it's sigma 1 Hermitian sigma Hermitian times U Hermitian. And now I can reinsert a V Hermitian B in between these two matrices and write it as U sigma 1 Hermitian V Hermitian V sigma 1 Hermitian U Hermitian. And this itself is basically B Hermitian. If B is this, then B Hermitian is U sigma 1 Hermitian times V Hermitian. So that is B Hermitian. And this quantity here is just A Hermitian, A is U sigma V Hermitian. So A Hermitian would be V sigma Hermitian U Hermitian. So this is equal to B Hermitian A Hermitian, okay, which is the same as AB whole Hermitian. So AB is a Hermitian symmetric matrix. In a similar way, you can verify properties 2, 3 and 4. And so this matrix B that we defined here, it satisfies the four properties required by the Moore-Penrose pseudo inverse, okay. So we now know that there exists a Moore-Penrose pseudo inverse. Now the uniqueness is a little more complex. So suppose there are two different matrices B1 and B2 that satisfy all four properties, okay, 1 to 4. Then if I look at B1 Hermitian minus B2 Hermitian, that is the same as B1 equals B1 AB1. So for example, we know that. So if I take the Hermitian, okay, so B1 Hermitian is the same as B1 A B1 Hermitian. So B1, sorry, B1 Hermitian is B1 A Hermitian, B1 Hermitian. So B1 Hermitian A B1 Hermitian minus B2 Hermitian A B2 Hermitian. So that is the difference between B1 Hermitian and B2 Hermitian. But by property 1, AB is equal to B Hermitian A Hermitian, okay. So I have B1 Hermitian A Hermitian, which is equal to AB1. So AB1 B1 Hermitian minus AB2, similarly for this AB2 B2 Hermitian, okay. So this difference is equal to AB1 B1 Hermitian minus AB2 B2 Hermitian. So you can see that this is equal to A times B1 B1 Hermitian minus B2 B2 Hermitian. So the range space of B1 Hermitian minus B2 Hermitian is a subset of the range space of A, because you can write it as A times some matrix, so it will have to, these columns will have to lie in the column space of A. Similarly, using these properties one and three, if you look at A A Hermitian times B1 Hermitian minus B2 Hermitian, you can write this as AB1 A minus AB2 A, but AB1 A equals A and AB2 A equals A. And so that is A minus A, which is the zero matrix. And so what that means is the range space of B1 minus B2, any vector belonging to the column space of this matrix will lie in the null space of A A Hermitian. So this span of B1 Hermitian minus B2 Hermitian lies in the range space of A, it also lies in the null space of A A Hermitian. Now if a vector lies in the null space of A A Hermitian, that means A A Hermitian V equals zero, which then means that just pre multiplying by B1, B1 times A A Hermitian V equals zero. Now you should verify this that again I am just repeatedly using these properties. It means that I mean these properties mean that if I look at B1 A A Hermitian, this matrix here, that is the same as AB1 Hermitian times A Hermitian, which is equal to A Hermitian. So A A Hermitian times V equals zero, then implies that A Hermitian V equals zero. So that means that the null space of A A Hermitian is equal to the null space of A Hermitian itself, almost done. So the upshot of that is that the range, so these two are the same and the range space of B1 Hermitian minus B2 Hermitian is a subset of both the range space of A and null space of A Hermitian. Now if you recall that this is actually the null set, then you are done. So basically that is what the rest of the proof does is that if there is a Z which lies in the range space of A, then Z can be written as a linear combination of the columns of A for some W. And similarly if Z also belongs to the null space of A Hermitian, then A Hermitian Z equals zero and substituting for Z, I have A Hermitian A times W equals zero, which means that W Hermitian times A Hermitian AW equals zero, which means that AW equals zero, but Z equals AW, so Z must be equal to zero. So this intersection space contains only the zero vector. So the range space of B1 Hermitian minus B2 Hermitian is zero, which in turn means that B1 Hermitian equals B2 Hermitian or B1 equals B2. So that concludes the proof. So that's all I have for today. The next class, which is the last class of the course, we will briefly discuss least squares problems. So what we are working up towards is to figure out how to solve Ax equals B and in particular in the case where there may not be an exact solution. So those are called least squares problems. In other words, we will ask what is the, how do we find a solution to the problem minimize with respect to x, Ax minus p, l to know. And so all these pseudo inverses basically will show up there and help us solve this problem. So that's it for today and we will meet again on Friday.