 We continue the discussion on concepts of linear algebra and we are into the last two important aspects of the concept with respect to matrices. One is the rank and the other is decomposition and specifically we will talk of singular value decomposition SVD, which is commonly used in many pattern recognition concepts. We will define what is a rank and also see how the rank is very closely associated with the SVD. I will give you two definitions of the rank and we will just discuss two short examples of the method of how to calculate the rank of a matrix. So, one definition says that it is the rank of a matrix is the dimension of the column or row space of a matrix. I will give you the other definition and then discuss what is this column space. The rank of a matrix can also be defined as the largest order of any non-singular minor of a matrix. The rank of a matrix will say A. So, as you can see that the word non-singularity is coming. So, the rank is very closely associated with singularity of a matrix. We look at this definition first, which is talking about the column or row space of a matrix A. So, we are talking about an arbitrary matrix A of size m cross n and the column space or row space is defined by the number of linearly independent column vectors, which form the matrix A and it is also called the span of A. So, it forms a span or the column vectors if they span a particular space, which we talked about space sometime back and that is the dimension of that corresponding space is actually the rank. And we will actually look at this definition, take an example of how to compute a rank in very brief and then look at the alternative definition of what is the largest non-singular minor of A. Now, there are several methods to compute the rank. We will briefly discussed one such method to compute the rank of A and therefore, steps I am writing in brief reduce to row equal on form that means, this is the processing done to A to compute its rank. So, take A and reduce to or to reduce not the correct word convert I will probably say or put to row equal on form identify the pivot column and number 3 and number 4 number of pivot columns is the rank or we can say rank is equal to the number of pivot columns. So, it will take a small example to illustrate this. Let us take I will not work out fully I will just leave part of this as an exercise for you. If you take an example let us say as given here I am taking an example from the book will give the references for these at the end of the talk. So, this is a simple 5 cross 4 matrix 5 columns and 4 rows and first we actually start with the first column and try to reduce the leading diagonal to 1 if it is not so by suitable row column manipulation interchange and then the task will be to minimize these values of the leading elements in the corresponding rows to zeros. So, just to give an example that what you can do is multiply this by 2 and subtract from this. You can directly subtract this from this row and sorry add these 2 rows basically and the 4th row is to be subtracted from the first and that is what will give you the leading diagonal. So, if you do that what we are saying is that these are the operations let me write them for the row R 2 it is basically R 2 minus twice R 1 I will write it in short R 3 is to be assigned by just adding R 2 and R 3 sorry R 3 and R 1 and R 4 you need to assign by subtracting R 1 from R 4. So, if you do this I leave it as an exercise for you that you will get this matrix the first row remains the same then you will have a 0 here because you have multiplied this by 2 and subtracted from here and the rest will be 0 to 4 1 minus 1 you just work it out this example as I said before is given in a book. So, you can actually look at that. So, this is what has been done with the first column you keep doing the same thing with the you know remaining column. So, you can see that we already have 0 and 1 leading diagonal 1 here. So, to put the corresponding values 0 at this point here and this will involve operations such as R 3. So, what has to be done with R 3 you have multiplied this by 2 and subtracted from here. So, R 3 will be taken as R 3 minus twice R 2 and then R 4 you just add these 2. So, R 4 will be and if you do this the first row remains the same 1 0 minus 1 0 4 second row remains same 0 1 2 0 1 this 1 this will become 0 this will also become 0 this 3 and then finally, you have a 0 0 and check it out yourself that these terms almost cancel out till this point minus 2 and 6 because that is what you are adding. So, you continue on this form the final thing is what you need to do now is you have already a 0 here. So, you have to make this 0 and then convert that is what you need to do. So, what you do is basically I multiply this by minus 2 and add it to this. So, R 4. So, the next operation or the last impact operation will be R 4 is R 4 minus twice R 3 and then you have to do this that will give you I am not reproducing the first 4 rows because they will remain the same and what you will get is what are these 4 rows they are actually these 3 rows are from here is upper sub matrix is going to sit here. These 3 rows again I repeat will be the same the last row will come as this. So, once you have done this that means what you have got is these 3 rows at this point in the last row 0 you need to find out how many columns you have which actually. So, if you find it here that there is a 1 here you need to find out the basis. So, you need to find out how many columns have only 1 at the corresponding leading position. So, you will have 1 2 and 3. So, in fact it will be this column here another column here and this column at this position because this row will be triple 0 1 minus 3. So, you will have 1 2 and 3 the rank of well it is not written with a capital R the rank of this matrix A is 3. This is one trivial way by which you can compute the rank of a matrix of course, there are many other better methods to compute a rank and we will talk about that soon. Let us go to the other definition of the largest non-singular minor. Now, the rank of a matrix R can be talked about now in different ways with respect to a non-singular minor I mean the corresponding rank of a matrix A is considered to be that you find out if there is 1 non-zero minor in a matrix A and that becomes the rank that is 1 way we can have that and if you take any other minor which is larger than that particular non-zero minor that minor vanishes or it becomes singular. So, the rank of a matrix is the largest non-singular minor. How do you obtain a minor from a rank A? Basically, if a matrix A is of size say n cross n or m cross n you suppress one row and one column and get one minor which is one order less you can suppress two columns and two rows and so on and so forth. So, you can suppress k columns and k rows for a matrix A and create the corresponding minor. So, if you take this definition now the second one with respect to a non-singular minor that means if a matrix is singular that means a determinant vanishes and it is real we are talking about definitely a square matrix here when you talk of a determinant or so you are talking about an n cross n matrix a square matrix which is singular determinant vanishing you are definitely talking about the rank of the matrix A in this particular case is less than n that means it is not a full rank matrix its rank is at least one less than its corresponding order and if you have an arbitrary matrix m cross n right here maybe we need to rub the board the rank of A is less than or equal to minimum of m comma n that depends upon the column space or the row space we talking about from that you find out the dimension of the column or row space of A it will tell you the corresponding rank. So, these are corresponding alternative definitions of rank and there are other terms substituted with rank it is we often call a singular matrix as a rank division matrix and corresponding to the rank we have a short of a rank space and a corresponding null space which is used in various manipulations of pattern recognition systems and applications. There are few properties I would like to specify with respect to the rank and to do that I will rub this example here as for example if you take the product of two matrices A and B and the corresponding rank of a matrix is less than or equal to minimum of rank of A and rank of B less than or equal to the corresponding ranks. Then there are other types of properties with respect to the ranks a rank of A is also the rank of its transpose of that corresponding matrix or if the matrix is complex we talk of the complex conjugate in general or also it is or equal to these are nice properties which one can exploit with respect to rank of a corresponding matrix and these are important A transpose A and A transpose are very common expressions you will get in certain applications of computer science and theories related to that. So, after we have got the concept of a rank we move to the important concept of analysis of matrices which is singular value decomposition or in short we will mention it as SVD people usually will call it SVD which means singular value decomposition and it is very closely associated with the concept of rank which we have discussed just now. SVD is only one of the type of decomposition possible for a matrix A we will just name a few other decompositions which exist in literature and definitely some of them are used in the field of pattern recognition SVD being the most commonly used popular one, but given a matrix A there are various other types of decompositions which are possible which are known as SVD type and just name a few of them the famous one of the first one which you will find commonly in the book is splitting a matrix A into two components lower triangular and upper triangular matrix a very common process which is used for many processes including equation solving you can also split this into another form which is L D U this equal to does not mean that this L N U is the same as this which I mean as that you can either split an A into L into U or L D and U where D is actually a strictly a diagonal matrix D is strictly a diagonal matrix L is an upper triangular matrix. So, lower triangular matrix and U is an upper triangular matrix I repeat L is a lower triangular matrix U is an upper triangular matrix in the special case of L D U unlike L U you should have once in the diagonal of L N U which is not guaranteed in the case of L U you can check this few things this other thing which is possible is actually very closely associated decomposition which is C C transpose which is basically the same matrix C is basically a lower triangular matrix C is a lower triangular matrix. So, C has a form as same as L and since it is a transpose it will actually give you a corresponding upper triangular matrix as well the other type of decomposition which is possible is Q multiplied by R where Q is a orthogonal matrix and do you know what is R? R is upper triangular matrix. So, it is an orthogonal matrix multiplied by an upper triangular matrix this process is actually called you know the name this is the Gram-Smith process or Gram-Smith process of orthogonalization this also has a name C C transpose an idea is actually called the Cholesky decomposition or factorization. So, we are talking different methods of matrix decomposition or matrix factorization. So, these are different things which are possible and of course you can also have similar to Q dot R is equivalent to Q dot H where Q is orthogonal and H is it is a positive definite matrix. So, where H is positive definite and finally for the SVD we have the famous decomposition it gives you U this is singular value decomposition in some sense it is similar to this, but we will tell you what are these U and V. So, I will write it in this particular form where if you have an m cross n matrix A I am following m cross n because we have followed that m cross n uniformly. So, the but of course in some books you may get p cross Q and things like that. So, you have a m cross m U which is not an upper triangular matrix do not confuse this with U in fact I will probably rub this. So, that we do not have any confusions with the notations for SVD compared with this these are some of the popular commonly used decompositions which are possible none of them is of SVD type lower upper L du Cholesky Gram-Smith and the Q multiplied by it. So, I will rub these U is an orthogonal matrix I will complete the expression then write. Then you have a diagonal matrix S or sigma as it is called sometimes and this is m cross n and then finally you have the V transpose which is m cross n. Out of these U and V are orthogonal matrices in various properties of matrices we must have mentioned concept of orthogonality. So, I have a look at that in fact some books will mention this that the basically these are orthogonal matrices which come consist of the corresponding eigen vectors and sometimes you some notations says that you consist of left eigen vectors V consist of right eigen vectors and this is strictly diagonal. What is the property of orthogonality? First of all U U transpose or it is inverse this transpose of a matrix is inverse if it is orthogonal the same property is this and what does U and V contain? Basically you are talking about U they are orthogonal matrices and so U consist of what we call as eigen vectors of a transpose and V consist of eigen vectors of A transpose A or the columns of U are the columns of U are eigen vectors of A transpose columns of V are eigen vectors of A transpose A and this singular values. So, if you write this matrix is a diagonal form it will look something like this some m of the certain size it is strictly diagonal and this consist of what are called singular values of A is what singular value has a very close relationship with the eigen values in fact they are the eigen values of A transpose or A transpose A which are you will find here. So, you can confine the corresponding eigen values of that and the root over of that will give you the singular values. So, how do you compute let us take an example and do this example keeping the definitions in the middle I will start on the left hand side of the board and move to the right hand side to compute just to show an example how this is done of course I must tell you that if you have a very large matrix computing the eigen vectors eigen values are doing an S V D decomposition there are good algorithms which will do this efficiently for you there are various libraries in C function programming languages like Mathematica and Matlab will also have each function libraries which does an S V D of an arbitrary matrix. So, but for the sake of illustration which we can work out in a classroom we will take a small example of a 2 cross 3 matrix let us take some values again this is from the book. So, you have to compute this three factors and for both of these as well as the eigen values here you need to compute A transpose or A transpose A given A you know what is transpose I am directly asking you to obtain the product of A multiplied by A transpose work it out you will find that now this is a 2 cross 3. So, A transpose will be 2 cross 2 matrix please work it out you should get a value matrix which has these values its elements. So, that is A transpose will come back to A transpose A. So, for this particular matrix there will be 2 corresponding eigen values we did this in the last class trying to find out the eigen values and eigen vectors given a matrix the 2 cross 2 the corresponding eigen values are you should be able to work it out yourself is on the method which we have done there at lambda 1 equals 10 lambda 2 equals 11 or 12 12 and the corresponding eigen vectors I want to write this in this form what is v 1 v 2 corresponding eigen vectors of this matrix corresponding to lambda 1 and lambda 2. So, these are 2 columns what do you get if you work it out you should be able to get this the corresponding eigen vectors corresponding to lambda 1 which is 10 will be 1 1 this will be 1 minus 1 for lambda 2. So, what it means this is corresponding to lambda 1 this is corresponding to lambda 2 the only thing which I have not shown you here is that although I am writing this v 1 v 2 let me correct myself a bit you will get these 2 as the corresponding eigen values let me order it in the sense that the largest eigen value I putting as the first eigen value the largest value. So, let us put 12 here and let us put 10 here. So, just a small correction here the eigen values have not changed, but I have ordered it that the first eigen value is the largest one out of the 2. So, if you have an n cross n matrix of a transpose a you will have n eigen values please order the eigen values in descending order starting with the largest value. And so for the corresponding largest eigen value 12 this is the eigen vector v 1 for lambda 1 which is this and the corresponding this order has to be preserved with respect to the singular value decomposition. Now to from this will actually yield u which is done by a Gram-Smith process or Gram-Smith process of orthogonalization and I will leave this to you as a self study please do not due to time constraints this is not a full-fledged course on linear algebra. So, we are giving the bare minimum. So, 1 by root 2 1 by root 2 and this will be again 1 by root 2 minus 1 by root 2 this from here to here by Gram-Smith process of orthogonalization. So, let us talk about v. So, given this a on the left hand side given this as your a what will be a transpose a will it be the same as a transpose will not necessarily that has a certain condition it will not be anyway because it is a m cross n matrix m is not equal to n. So, this will be a 3 cross 3 matrix and if you work out the elements in order for the sake of time let me give you the values it is 10 0 2 second row is 0 10 4 and it is a 3 cross 3 matrix a transpose n. So, we have 3 eigen values lambda 1 lambda 2 lambda 2 I will put them in descending order as we did as I talked about earlier. So, 12 this will be 10 and lambda 3 will be 0 basically saying that this is a rank deficient matrix not full rank you do not have all the eigen values which are non 0 here. Now, not necessarily when you compute you will get it in this order you might get the first eigen value as this, but we are ordering them as per descending order and then the corresponding eigen vectors for this. So, I am writing them as v 1 sorry v 1 v 2 v 3 corresponding to v I am not writing this as v because you have to do a Gram-Smith process. So, the corresponding eigen vectors for 12 10 and 0 you will get it as this check it out yourself. Which by the help of this Gram-Smith process will give us the v and I am actually giving the v transpose remember Gram-Smith of this will give you the v and you do a transpose to get this I am writing the v transpose directly and can you give me the values what is the first row you will get 1 by then the second one root 5 v transpose I am writing directly what is the second row 2 by root 5 0 root 18 then minus 5 by you can use any algorithm to compute this. Now, you can see that the A transpose A A transpose and A transpose A look at the eigen values you have I did say sometime back where you will get them same except that now you have 1 which is equal to 0 that is the least one which you have for m cos n the first two are same. So, the corresponding now if I write the value of s. So, now the calculation of the sigma sometimes in some books you will write it as s. So, you will write this is the corresponding eigen values singular values are the root over of the eigen values of the A transpose. So, you will get them as root over 12 root over 10 that is all and this is a diagonal matrix I did say, but of course what you will get is if m is not equal to n you will get a non square diagonal matrix as well which having one column or one row as 0. So, the final form of the SVD I am not adding it together will be u sigma and v transpose or u is s and v transpose at this called the eigen vectors left eigen vectors then you have this matrix. So, if it is original matrix is of 2 cross 3. So, you have 2 cross 2 then 2 cross 3 here which is the middle mat and the right hand side was here which is 3 cross 3. So, this is an example of a singular value decomposition which will be used extensively in many pattern recognition algorithms which you will see. So, this concludes the discussion on introduction on linear algebra and concepts which are necessary for pattern recognition applications. We move on to further concepts of mathematical concepts of pattern recognition also I must remind you that you can find relevant material about linear algebra and its applications in a set of books which are given in the next slide which is coming up for you. Thank you.