 So the last time we were looking at properties of matrix norms, this is the long chapter in the textbooks, so we will be discussing this for a couple of more classes. So specifically, we discussed L1, L2, and L infinity norm, and these are three norms that are different from the L1, L2, and L infinity norm that we are going to define in this class. These are basically matrix versions of vector norms, and in particular, the L1 norm is the sum of the magnitudes of all the entries of the matrix, and that is indeed a matrix norm. The L2 norm we defined in the previous class is the sum of the squares of the entries of the matrix to the power half, and that's also known as the Frobenius norm. The L infinity norm, if you do n times the maximum magnitude entry of all the entries, the largest magnitude entry among all the entries of the matrix, then that is what we call the L infinity norm, and that is also a matrix norm. But these are three different norms compared to what we are going to consider in this class. So we discussed about induced norms, and the point is that you can start with any vector norm and this could be any vector norm, and then if you define the matrix norm to be the maximum value that norm of Ax. So Ax is a vector here, so you're taking the vector norm of Ax, and you look at the largest value this can take over all x such that for the same norm, norm of x equals 1. Now that quantity, of course, it's non-negative and it turns out, in fact, we stated and proved the theorem to that effect. It turns out that this is a matrix norm. So these are called induced norms because there is a vector norm, underlying vector norm, that is inducing a matrix norm, and we started discussing about examples of induced norms, and today we will continue and discuss one or two more examples of such induced norms, and then move on to discussing about the spectral radius and its properties. Okay. So just to recap what we discussed at the very end of the previous class, we were discussing about induced norms and we particularly discussed this maximum column sum norm, okay? And this is what I'm writing with, three bars and one next to it, and this is defined as the maximum column norm among all the columns in A, in the matrix A. Written in terms of the entries of the matrix, what you have to do is to take the sum of all the entries of a given column, okay? The magnitude entries of a given column, say the jth column, and look which column among all one to n columns gives you the largest value, and that's what we defined to be the maximum column sum norm of the matrix A. And there are two ways to show that this is indeed a matrix norm. The first way is to go from first principles, start with the definition of the matrix norm and show that it satisfies the four properties we need for something to be a matrix norm, namely the non-negativity, positivity, homogeneity, triangle inequality, and submultiplicativity. That's five properties, but non-negativity and positivity are lumped as the first property. That is the first way. The second way is to show that this norm, whatever dysfunction that we're writing here, is induced by some other vector norm. And so in other words, what we want to show here, so the claim is that this norm is induced by the vector L1 norm. So we need to show that whatever we've defined here is in fact equal to the maximum of the L1 norm of AX over all vectors such that the L1 norm of X equals one. There are multiple ways to show this, but here is one easy way. So if A1 to An are the columns of A, then clearly by definition, the maximum column sum norm of A is the maximum among all one to N, I going from one to N, of the L1 norm of the ith column of A. Now, if you define a vector X1 to XN, and you look at what norm of AX L1 is, that is just expanding it out in terms of the columns, it is X1 A1 plus et cetera up to XN An, L1 norm. Then I use a triangle inequality of the L1 norm. The L1 norm is a vector norm, and therefore it satisfies triangle inequality, and I take the norm inside the summation. So I'm left with, so I get a less than or equal to, summation I equal to one to N, the L1 norm of XI AI. XI here is just a scalar, so I can bring that out and write it as mod XI times the L1 norm of AI by the homogeneity property of the vector norm. And then I can further up about this by replacing all of these guys, norm AI L1, with the maximum value, which is the max one less than or equal to K less than or equal to N, the L1 norm of AK. Now this has no longer depends on I, so it can come right out of the summation. And the summation I equal to one to N mod of XI is nothing but the L1 norm of X times, so and then this quantity here is by definition the maximum column sum norm of the matrix A. So the upshot of all this is that if I'm trying to maximize the L1 norm of AX, subject to the L1 norm of X equals one, I can upper bound that by the maximum column sum norm of the matrix A. So this we called as expression A. Then conversely, if you choose specific values of the vector X, specifically if I take EK, EK is the vector with zeros everywhere, except a one in the Kth position or you can also think of it as the Kth column of the N cross N identity matrix. Then if I look at the maximum of this, this EK is a vector which satisfies the constraint. The L1 norm of EK is always equal to one, as you can see from here. And so this maximum, I'm looking across all possible vectors that has to be at least equal to its value at one particular point, which is EK. And so if I substitute EK, then that will only pick off the Kth column of A and I'll be left with the L1 norm of AK. So the maximum overall X such that L1 norm of X equals one of the L1 norm of AX is at least equal to the L1 norm of AK for K equal to one to N. So since this is a lower bound for all K, it's of course, if I take the maximum of these numbers overall K from one to N, that will still be a lower bound because N is a finite number here. So we have that the maximum L1 norm of X equals one of the L1 norm of AX is at least equal to the maximum of the L1 norm of the columns of A, which again by definition is the maximum column sum norm of the matrix A. So this is what we called inequality B. So what have we shown? We've shown that this maximum value, whatever this is, is at least equal to the L1 norm of A and at most equal to the L1 norm of A. So if that is the case, then it must mean that the maximum of this, the L1 norm of AX subject to L1 norm of X equals one must be equal to the maximum column sum norm of the matrix A. And thus, this norm that we just defined is indeed induced by the vector L1 norm. And therefore, A L1 is a matrix norm. So the exercise that I closed the previous lecture with was to show from definition that this L1 norm of A, the maximum column sum norm of A is indeed a matrix norm. Okay, so now we'll continue with a couple of more examples. This is the maximum row sum norm. So this is very similar to the maximum column sum norm except that instead of taking the L1 norms of the columns of A, we'll now take the L1 norms of the rows of A. So if we write A infinity, so I'm fixing an I, so that's fixing a particular row of the matrix and I'm taking the L1 norm of all the entries in that row. And then I'm looking for which row from among the rows one to N gives me the biggest value and that's what I'm defining to be A infinity. You can see that this is actually equal to the maximum column sum norm of A transpose. So by that itself, you can see that this is matrix norm but it is also induced by the vector L1 norm, by the vector L infinity norm and therefore it is a matrix norm. Showing this is very similar to the column sum norm so I won't do that here but we'll just move on directly to the next norm which is the spectral norm. The spectral norm is by far the most important norm that we will look at in this course and we write it like this and it's defined as the A2 is equal to the maximum square root of lambda such that or the value lambda is an eigenvalue of I'll write it for the complex case but of course if it's real, this Hermitian will be replaced by transpose but it's A Hermitian A. So we haven't formally discussed eigenvalues but unfortunately for many of these results we will need the notion of eigenvalues but we'll connect all of these together later in the course when we discuss eigenvalues also. So basically lambda is a quantity that satisfies A Hermitian A x equals lambda x for some x not equal to zero. And so one thing to note here immediately is that if I pre-multiply this by x Hermitian I have x Hermitian A Hermitian A x is equal to lambda is just a scalar so I can pull that out and write it as lambda x Hermitian x, okay? And this quantity x Hermitian A Hermitian A x is actually equal to the L2 norm of A x square. So we've seen that already that if I have a vector y, y2 squared is equal to y Hermitian y. This is another way to write the, it's just the sum of the magnitude squares of all the entries of y but we can also write that as y Hermitian y. And so this quantity is real and positive and this quantity is also real and actually non-negative. So both these are real and non-negative and so you cannot suddenly have lambda being a complex number. And in fact, since both are real and non-negative and in fact since x is not equal to zero this is strictly greater than zero and this is greater than or equal to zero which implies that lambda is always non-negative. So basically square root of lambda is always real and non-negative. Now this particular norm is in fact induced by the vector L2 norm so that A L2 square is actually equal to the max over all x L2 equals one of A x L2 square. So this is the spectral norm. So these are the three examples I wanted to discuss about induced norms. So the maximum column sum norm, the maximum row sum norm and the spectral norm. We'll discuss the spectral more later but first before that recall that for vector norms we know that if you're given a norm and a non-singular matrix, if you define a new norm to be the norm of, okay, so let me just write that. You've seen this property before. If is a vector norm and A is non-singular then if I define this to be the vector norm of A x, then this is a vector norm also. So basically given, if we know of a particular vector norm then given any non-singular matrix we can define a new vector norm. Similar, there's a similar result for matrix norms. So we have this theorem. If is a matrix norm on the space of n cross n matrices and if S is non-singular, then if I define A S to be the matrix norm of S inverse A S for any A in C to the n cross n, this quantity A S is a matrix norm. So this quantity S inverse A S is what is called a similarity transform on the matrix A and it has lots of very nice properties which we'll actually study in quite detail later on. But for now we're just observing that if you're given a matrix norm and any non-singular matrix, you can define a new matrix norm using that non-singular matrix. So how do you show this? The proof is very simple. Of course, the properties like homogeneity, non-negativity, positivity and triangle inequality directly follow from the properties of this matrix, this matrix norm here. And so the only, interesting thing we need to show is the sub-multiplicativity. So for example, I mean, just to make my point, just to make my point, so if you had, if you took take A plus B and then you're looking at this S norm, this is equal to the matrix norm of S inverse A plus B times S inverse A plus B. A plus B times S, which is equal to the matrix norm of, I can take this S and S inverse inside the brackets, so that it's S inverse A S plus S inverse B S, which now this is a matrix norm, so it will satisfy the triangle inequality. And I can write it as S inverse A S plus S inverse B S, which is actually equal to the S norm of A plus the S norm of B. So it satisfies triangle inequality. So how about the sub-multiplicativity? That's also very easy. So if I look at A B S norm, then that is equal to the matrix norm of S inverse A B S. And I can just insert an S S inverse in between here because S S inverse is just the identity matrix. So this is equal to S inverse A S S inverse B S. Now I use the fact that this is a matrix norm and so it satisfies the sub-multiplicativity property. So it's less than or equal to S inverse A S times the matrix norm of S inverse B S, which is nothing but the S norm of A times the S norm of B. So we now know several ways of coming up with different, different matrix norms. One way is you start with any vector norm that you know and then look at the induced norm. The other way is start with any matrix norm you know and consider a convenient non-singular matrix and then you do S inverse A S and take the norm of that and that becomes a new norm. So there's a great richness in the types of or number of different types of norms that you can construct.