 In this lecture, we will be discussing solutions of systems of linear equations. First, we will discuss the abstract ideas regarding nature of solutions. Then, we will go into the basic idea of the methodology by which we solve such systems of linear equations. And then, we will consider the particular case of homogeneous systems of equations in which the right hand side is 0. The other case when the right hand side is not 0 will be taken up later, which is the more regular and routine case of such systems. After this, we will consider two important issues in such systems. One is puroting and the other is partitioning. First, nature of solutions of systems of linear equations. The systems of linear equations in general can be expressed in this manner A x equal to b in which the matrix A is the coefficient matrix. And when we enhance it or augment it with the right hand side vector on this side, then we construct this larger matrix or augmented matrix, which is A and B together. We will refer to this matrix, this larger matrix as the augmented matrix. The first question that arises is does this system have a solution? That is the question of existence of solutions or consistency of the system of equations. If the solution does not exist, then we say that this particular system is inconsistent. That is there is some internal conflict among the different equations in the system. Now, when we try to answer this first question that is whether this system is consistent or whether solution exists, then we have this following important theoretical result that is A x equal to b has a solution. This statement is equivalent to this second statement, which is that b is in the range of A. And that is again equivalent to the third one, which is that rank of A is the same as rank of the augmented matrix. This is very easy to visualize, because if we write this system A x equal to b in this manner, in which the columns of A are written as A 1, A 2 and so on, then the entries of x will appear in this manner. This shows that what we are talking about is actually a linear combination of these columns on the left hand side, because A 1 into x 1 gives us this. Thus column A 2 multiplied with x 2 gives us this and so on. Now, when we say A x equal to b has a solution, that means that there is a set of x 1, x 2, x 3 values, which when put in this linear combination will result in a total sum which is equal to b. And that is as good as saying that b is in the range of A, that is as a linear combination of columns of A as A x, A multiplied with x, we can find b, that is directly from definition. And this is also equivalent to say that since as a linear combination of these columns A 1, A 2, A 3, A 4, etcetera, we can find b, that means when we enhance these columns with one more column, then this is a linear combination of all these. And that means this is linearly dependent on these, then that means that whatever rank these vectors together have, whatever rank this matrix has, this will not add anything new in terms of linear independence, that is why the augmented matrix also has the same rank. So, when the rank of the coefficient matrix and the rank of the augmented matrix are same, in that case b will be in the range of A. And that means that system is consistent, there is a solution, the question is still open whether there will be only one solution or many solutions. And then we come to the second question which is that of uniqueness of solution, for that the important result is that when these two ranks are same and also equal to the number of columns of the matrix A, there is number of variables, then that will imply and will be implied by this statement which is that the solution of A x equal to b is unique, which is again equivalent as saying that the corresponding homogeneous equation, homogeneous system A x equal to 0 as only the trivial or 0 solution. Now, to verify, to establish the result as shown here, we consider this situation, that is suppose p and q are two different solutions of this system A x equal to b, p and q are two different solutions. Then we can say A p is b as well as A q is also b and in that case we can subtract and find A into p minus q is 0. And what is A? It is a matrix with A 1, A 2, A 3, etcetera as columns and what is p minus q? p minus q is a vector with entries p 1 minus q 1, p 2 minus q 2 and so on. So, when we open this in this manner, then we find the first entry of p minus q into the first column plus the second entry into the second column of matrix A. And so on, this is equal to the 0 vector. Now, we know that since the matrix A has full n rank and it has only n columns, that means all these n vectors are linearly independent. Now, if A 1, A 2, A 3, etcetera are all linearly independent, then from the definition of linear independence, we know that this sum equal to 0 will mean that individually these coefficients are all 0. That means p 1 minus q 1 is 0, p 2 minus q 2 equal to 0 and so on. That means p 1 equal to p q 1, p 2 equal to q 2 and so on which will mean that the full vector p and the full vector q are same. That means these two solutions that we picked up, they cannot be different. They have to be same if A 1, A 2, A 3, A 4, etcetera are all n linearly independent vectors. That shows that if these two have the same rank and that rank is equal to n, that will mean that this system has a unique solution. It has a unique solution for all b in particular that is true for b equal to 0 as well and that shows that A x equal to 0 also has a unique solution. We already know that x equal to 0 certainly satisfies this equation system, whatever may be the A matrix and therefore, in that case it will mean that it has its unique solution is 0 solution or the trivial solution. Again from here, we can show this in a similar manner and that will that again from the definition of linear independence that if this has the trivial solution only then that will mean that rank A is equal to n and so on. That way we find that this result regarding uniqueness of solution and this earlier result regarding existence together gives us the complete theoretical background for the nature of solutions. When it happens that this rank and this rank are same, but not equal to n, but somewhat less then the solution will exist, but it will not be unique. In that case, we will have infinite solutions for the system and then we need to characterize those infinite solutions. How do we characterize that? Suppose in a situation rank of A and rank of the augmented matrix is same and is equal to k that is less than n. In that case if x bar is one solution and x n is a member of the null space, which means A x n is 0 that is if x n is a member of the null space then x n get mapped to 0 and x bar is some solution of the original system given. Now if we add these two then we find that this also turns out to be a solution. On this side we have added these two, on this side this will be b because A x n will contribute nothing that is 0 only and A x bar contribution will remain. That means for every member of the null space x n if we add that to another to a solution of the original system then this sum is also going to be a solution of the original system and since the null space has infinite members so adding any of them we will get yet another solution. So, that will characterize the complete solution of the linear system A x equal to b. So, now we have two sub problems. One is to make a description of the null space to describe all vectors which can sit in this place and to find one solution of the system A x equal to b and then when we can we add that one solution one particular solution with this null space member arbitrary null space member we have got a complete description of the solution of A x equal to b. So, in the basic idea for solving the system we have three small pieces of work. First to diagnose if the system does not have a solution we have to diagnose the non existence of a solution. If the solution exists then we have and if that in that case if it is unique then we have the problem of determining that unique solution that is an easy job comparatively difficult will be the third piece of work which is if the system has infinite solution. Then our task will be to describe these infinite solutions to diagnose the non existence to determine a unique solution and to describe infinite solutions these are the three pieces of work. In all these cases it will help for doing all of these a prior you do not know which will be the case to do any of these or all of these we take the path to decouple the equations to the extent possible using elementary transformations which we discussed in the previous lecture. What we do to solve this or to simplify this is that we take a x equal to b and apply suitable elementary row transformations on both sides. On this side it will be applied on on the left side it will be applied on a on the right side it will be applied on b. That means every elementary row transformation that we apply on both sides is equivalent to the pre multiplication by the corresponding elementary matrix say first with R 1 simplification to an extent. Next we apply another row transformation in order to simplify the situation further say R 2 both side then R 3 then R 4 and so on. Finally, lot of elementary row transformations get applied that means a lot of pre multiplication take place with the corresponding elementary matrices. Finally, we can consider this entire product R q R q minus 1 R q minus 2 up to R 2 R 1 a this whole thing together is written here a is written as it is and the product of all these elementary matrices is sitting as R that same R will be sitting on the right side also as R b. Now, after this point we examine what happens to this matrix here R a we want R a to be as simple a matrix as possible as close to an identity matrix. Accordingly we decide our strategy on finding the suitable row transformations that we apply on both sides. In the best case with complete reduction R a becomes identity and identity into x is simply x and in that case whatever is the vector on the right hand side the entries of that will give us the values of x 1 x 2 x 3 etcetera. If it cannot be reduced up to identity then we try to make as much simplification as possible in the matrix R a. So, that finally, a little bit of processing is sufficient to find out the values of x x 1 x 2 x 3 etcetera entries of x or to describe them in terms of a few variables in the case of infinite solutions. The process as we discussed earlier has two issues one is to describe the infinity of infinite solutions in which case we need to find the null space of a. On the other hand to find one solution the problem of finding one solution we will consider in the next lecture the immediate task is to characterize or describe the case of infinite solutions in which the matrix a has a non-trivial null space. So, for that purpose let us first see the elaboration of the scheme which we utilize and then we will be talking about the formal structure of the method. For that we take this problem of finding the solution of this system in which a is of size 5 by 7 it is a rectangular matrix. Now, when we say that we want to find the solution of this system x equal to 0 in which the right hand side is 0 that is equivalent at saying that we want to find out the null space of a because we are looking for all those vectors x which in this product x give us 0. So, we are basically finding trying to find the null space of a. Now, note that if we apply an elementary row transformation on a on this side then on that side we will be applying the same elementary row transformation on 0 and whatever elementary row transformation we apply on the 0 vector it will still result in 0. That is why it is not necessary to go on writing the 0 on this side we will just keep on applying elementary row transformations on a with a view to reduce it significantly. And here what we I will do is that at every step whichever values in that 5 by 7 matrix get changed I will show that with a tick mark like this and whichever values it will remain same I will show as dots. And whenever it is needed for us to use a particular value a particular entry then we will give it a name otherwise if that particular entry becomes 1 or 0 then we will write that 1 and 0. That means we will be using symbols which are 1 when it is known that it is a 1 there 0 when it is known to be 0 there. Tick mark when it is changed from the previous step and dot when it remains unchanged from previous step and we will use a symbol when that is going to be used. So, these 5 notations we will be using in elaborating this method. So, first we have all static mark 5 by 7 matrix our first step to simplify this matrix is to find a 1 here for that we examine whether this number is 0 or not. If it is 0 then we would like to interchange rows to get a non-zero number here either from here or here or here or here. So, first we see whether this is 0 or not in the first case suppose we find that it is not 0. So, give it a name for this entry for the time being we give a name a and then we wanted a 1 here to apply an elementary row transformation which is divide the first row by a that is this number. If we divide then what we get we get 1 here something here whatever is all the old entry by a. So, something something these get changed the other 4 rows do not get changed. So, these do not get changed our next intention will be to get 0s here right now what are there. So, there are 4 numbers sitting here let us give them some names how do we get 0s here from elementary row transformation what we do is that we take the first row multiply it with b and subtract from the second row. Then whatever happens in the later columns later entries one thing is sure that here we will get 0. So, next elementary row transformation will be a set of 4 transformations and those 4 transformations will be from second row multiply subtract from the second row subtract b times the first row. So, b minus b into 1 will give 0 here and other entries will appear here in the meanwhile the first row remains unchanged and here we will get 0. So, second row will become second row minus b times the first row and so on. Similar situation for the third row fourth row fifth row. So, all these rows will get changed here we will have 0 and here whatever 0 0 0 this will be changed currently we are not interested what are those change numbers right. After processing the first column like this and getting 1 0 0 0 0 now we shift our attention to the second column we want a 1 here. Here we cannot do anything because right now we cannot do anything here because 1 is here. So, we preserve this 1 below that the 0 that we have found we preserve we want a 1 here if possible. Then we examine what is this is it 0 or is it non-zero. So, suppose in this case we find that it is non-zero call it here in a later situation we will consider 0 there and then we will examine what to do if a 0 appears in such a situation. So, currently suppose it is f right we wanted a 1 here. See if it is 0 then we cannot get a 1 here by just dividing because 0 with whatever you divide you will get 0 only. So, if it is a non-zero number then we can divide by this number to get a 1 here right. So, the next eliminate row transformation will be divide the second row with f. As a result the first row will remain unchanged the second row will change 1 will come here and then other stuff will come here and the lower rows will remain unchanged. After getting the 1 here we will try to get 0's in the other positions of this column that means here, here, here and here. What are there currently? Currently some numbers which have been copied from the last step. So, give them some names G H I J. Next what will be our steps to get 0's in these locations in place of G H I J? We will multiply row 2 with G H I J and subtract from row 1 3 4 5 respectively. In doing that we will not be spoiling this 1 and these 0's because this 0 multiplied with G H I J and subtracted from the respective rows will not lead to anything new because this is 0 here. So, as we multiply the second row with G H I J then we will get 0's here right. And now onwards to save time I will not be writing all this detail I will be just discussing the steps and carrying out the appropriate operations in the matrix. In the first column there will be no change because this 0 will fail to make any changes here that has been part of the strategy right. In the first column these entries will become 0 by R 2 equal to R 2 minus R 1 equal to R 1 minus G into R 2 and so on. So, we will have 0 then certain change entries this will remain unchanged in this step these will change. Now in the same process what we will be looking forward next after we have we have set these 2 columns we will be asking for a 1 here. So, for that we need to first see what it is what it is this can be 0 or non-zero what to do if this is non-zero that we have already seen we divide this row with this number. Now let us suppose that in this particular case this turns out to be 0 if this turns out to be 0 then we examine what is below that what is here suppose this is non-zero suppose this is non-zero set K then what we do is that this third row and the fourth row we interchange as we interchange the third row and fourth row then these are 0's nothing very special these entries and these entries interchange their locations and this K comes here this 0 goes here these will be changed they will be swapped. Now this is non-zero and we wanted a 1 here. So, what we do we divide the third row with K whatever is this number that changes these entries and here we get 1. Next in this column other than this 1 we want everything else to be 0 this fellow is already 0 other 3 we want to be 0. So, what are these numbers by the way suppose they are l m n currently these are currently l m n then what we do we subtract from the first row l times the third row subtract from the second row m times the third row and subtract from the last row n times the third row that way we get these 3 also as 0 first second and fifth have changed. So, in this step these have changed and these have remained same after coming this point then we examine what is this number right suppose this is found to be 0 then we examine what is this number in this case suppose this also found is found to be 0. That means that in this column we could not get a 1 the way we found till now in this case we could not find that. Now we hunt for the next column what is here we examine this suppose this is found to be p then we want this to be 1 just like earlier case here we could not find unfortunately here within we have found a non zero number we want a 1 here and as such we will be dividing this row with p and that will mean that we will get to 1 here these will change and then we want a 0 here if we have got 1 here we want 0 below it and 0 above it all these locations we want to be 0s. So, currently suppose we have got here p q r s then fourth row multiplied with p q r and s is subtracted from row 1 row 2 row 3 and row 5 right. Then that will give us these four entries as 0 0 0 0 and 0 and these corresponding terms will change by whatever these will remain same in this step. Now after getting this till this point we examine what is this and what is this what is this when we examine if it is not 1 say suppose it is 0 and suppose this also found to be 0 then we cannot do anything about these numbers because above the 0 or below the 0 below the 0 it will be 0 only above the 0 whatever is sitting here we cannot do anything regarding this. So, now the non-trivial entries in this matrix remain in these locations in that column in which there is no 1 in the leading position. So, now let us give them names the non-trivial entries suppose we call them alpha beta gamma then next round of non-trivial entries are in these columns. So, let us give them also certain names say delta epsilon kappa lambda delta epsilon kappa lambda mu nu theta phi these are the non-trivial entries which remain in the matrix and now let us call this reduced matrix simplified matrix as a tiller from a to this step has been arrived at through all elementary row transformation. And now in this reduced matrix see this structure like this like this like this like this like this like this like this below which everything else is 0 this form is known as the row reduced echelon form echelon is a French word which means steps. So, this step wise formation that we have got through row reduction this form is called the row reduced echelon form or RREF. So, through a series of elementary row transformations we have reduced the original matrix a to this form a tiller and this form is the row reduced echelon form. The features of this form are that the first non-zero entry in any row is a one in this row there is no non-zero entry if it were there it would be a one wherever it appears. This particular entry is called the leading one. So, the first non-zero entry in any row is a one which is quite often referred to as the leading one. The second feature is that in the same column as the leading one other entries are 0 wherever that leading one appears in that column other entries are all 0 above and below above and below above and below. The third feature is that non-zero entries in a lower row appear later. The non-zero entries in this row will appear later than in this row and so on. So, with these three features we have got the row reduced echelon form. Now, again go back to that original theme that if a x equal to 0 is the system then the solutions of this system remain unaltered through all these elementary row transformations that we have carried through that means if this is 0 then that will also mean that a tiller x is also 0 which means that we can say the equations are now greatly simplified. From the first equation we find that x 1 plus alpha x 4 plus delta x 6 plus mu x 7 equal to 0 and therefore we can express x 1 as minus alpha x 4 minus delta x 6 minus mu x 7. Next the second equation tells us 0 into x 1 plus 1 into x 2 plus 0 into x 3 plus beta into x 4 and so on. So, keeping x 2 on one side we take everything else on the other side and then we get minus beta x 4 minus epsilon x 6 minus mu x 7 like this. From the third row we similarly get x 3 as minus gamma x 4 minus alpha x 6 minus theta x 7 x 4 we write simply as x 4 because the fourth column does not have a leading one. So, that we leave then x 5 that is found from here this is the leading one of the fifth column. So, x 5 we get as minus lambda x 6 minus 5 x 7 then x 6 we do not have to write like that because for the sixth column the leading one is missing this column does not have a leading one. So, x 6 we write as x 6 only x 7 we write as x 7 only. What do you notice here all the variables in particular variables x 1, x 2, x 3 and x 5 have been expressed in terms of three variables x 4, x 6 and x 7. That means that the variables x 1, x 2, x 3 and x 5 those we write as x 1, x 2, x 3 and x 5 with a leading one in the corresponding column have been expressed in terms of those columns which do not those variables the corresponding columns of which do not have a leading one. And that is what we do as a rule variables corresponding to columns having leading ones are expressed in terms of the remaining variables. And this whole thing can be written nicely in terms of this matrix vector multiplication which will look like this x 4 is simply x 4 1 into x 4 plus 0 into everything else x 6 and x 7 are similarly 1 into x 6, 1 into x 7 nothing else. The other variables are expressed in terms of x 4, x 6, x 7 that means that this system will have a solution as any vector x in which these three variables can be chosen at will and after the choice of these three variables rest of the variables get defined by this. So, for that matter there is no more any reason to call them x 4, x 6, x 7 we can call it simply u 1, u 2, u 3 or anything any name. And this matrix this 7 by 3 matrix inside itself actually codes the null space of the matrix A which is the same as a null space of the matrix A tilde. This first column is one null space member, this second column is another null space member, the third column is the third null space member and these three are linearly independent. That linearly independence shows here, in the third one there is a one in the seventh location which cannot be found by any linear combination of these two. Similarly, here a one is found in the second one which cannot be found by the any linear combination of the others because others have delos here. Similarly, here this cannot be found by any linear combination of the other two rows that shows that these three are indeed linearly independent. Any of these is a solution of these and arbitrary linear combination of the three by giving arbitrary values here will also be solutions of the equation x equal to 0 and this describes the null space of the matrix. So, these three columns are actually three linearly independent null space members. Any of them or any linear combination of them can sit in the place of x n in the general solution for system A x equal to b for which the same A is used. So, solution of A x equal to 0 will be then written as this matrix having the null space members z 1, z 2, z 3 etcetera. Here these three vectors are z 1, z 2, z 3 the null space members and how many of them will be there? In this case we have shown three that means this matrix has rank four. This has seven columns and if rank is four that we can see here that is after the reduction these four rows are linearly independent in terms of columns you can say first, second, third and fifth columns are linearly independent. So, four columns are linearly independent or four rows and not the fifth one. So, that means that rank is four and rank plus nullity will be the number of columns here. So, in this case the nullity is three that means null space will have three linearly independent members. So, in this case k is four n minus k is three and so you have three arbitrary values u 1, u 2, u 3 to choose and these vectors sitting here will give us a basis for the null space because null space has three linearly independent members and these are three such linearly independent null space members. So, they together form a basis for the null space. Any linear combination of these three through these coefficient arbitrary values that will give us a null space member. So, this is how we solve homogeneous systems of linear equations by reducing them to the row reduced issue of form. Now, what in this entire process as well as the process of solving square systems for a unique solution when that is the case. We use another important technique quite often which is called pivoting. In this example I have not discussed that business of pivoting any non-zero entry we were trying to get in the leading position in order to later divide it with that number to get a 1 here, but typically in numerical work what happens is that if that number is very small then division with that number may lead to numerical error which may turn out to be quite unacceptable at times. Therefore, there is a non-trivial exercise for pivoting. Note that this kind of steps typically try to get 1 at the diagonal or leading position with 0 as well. A key step is division by the diagonal or leading entry. Now consider a matrix in which suppose the leading k by k part has been already process and something is sitting here which is say delta. Now, if that delta is 0 then we cannot divide by in that case I have already discussed that we can exchange rows in order to get a non-zero number sitting here. So, we cannot divide by 0, but even if it is not 0, but if it is a number with very small magnitude say 0.000003 or minus 0.0005 something of that sort. So, in that case we should not divide by delta. Why we say we should not divide by delta because division by very small number may lead to numerical errors in a large computational program. So, cannot divide by 0 should not divide by delta that means by small number. Now, we can still apply row interchange or for that matter column interchange if we want to get a larger number here. So, what we do is that we try to apply row interchanges to get the largest here whatever is the largest in this column say big. So, to get that number here we apply a row interchange. So, that means if we do pivoting then whenever we were faced with a situation of dividing by this leading term rather than dividing directly with it we could hunt out the largest number from this point onwards downwards and put that number here through a inappropriate row interchange that would be the pivoting step. So, if we only apply row interchange to get this big in this location which will send this delta here then that kind of a pivoting is called partial pivoting. On the other hand there is also a technique called complete pivoting in which we first apply row interchanges and then column interchanges in order to get the largest possible number from this entire block to come here that is called complete pivoting. Now, quite often in computational programs we will find that complete pivoting is actually a wasteful exercise we might do it, but it does not achieve a lot of advantage quite often or most often partial pivoting helps to sort out the issue much more easily though complete pivoting can be done. So, complete pivoting does not give a huge advantage over partial pivoting, but requires maintaining of variable permutation because column interchange will mean that the variables x 1, x 2, x 3, x 4 actually get permuted. So, that extra book keeping you have to perform if you decide to do complete pivoting. On the other hand partial pivoting is much simpler to handle and it accomplishes the task to a satisfactory extent. On the other hand if the problem appears that this is extremely small and below that everything else is also extremely small all are 0 that will mean that you have a singular matrix and all of these you can consider as 0 and the case is actually one of this type of situation in which you should consider them as 0 and proceed to the next column. So, this step of pivoting will come across quite often in the next lecture as well in the current one one more small topic we need to discuss with in later theoretical developments will be quite often encountered and that is the issue of partitioning and block operations. Quite often you will find that a matrix can be written partition in this manner and along with that the variable vector also gets partition in this manner and the right hand side vector also gets partition in this manner and this is fine as long as the associated steps of matrix multiplication, matrix vector additions and the equalities make sense. What I mean is the following a large matrix A and a large long vector x and another long vector y may be forming a system A x equal to y in this manner and in this manner if we write it in broken form then it will mean A 11 x 1 plus A 12 x 2 plus A 13 x 3 equal to y 1 and so on. So, we will have the first row block that will appear will be like this and this kind of a partitioning is valid as long as these individual matrix vector multiplications make sense their dimensions are compatible. These additions make sense their dimensions are compatible and this equality makes sense on both sides we have vectors or matrices of the appropriate correct sizes. Similarly, there will be a second row block from here this kind of partitioning and block operations in that manner can be carried out over complete blocks and these operations help a lot in theoretical development and selective solution of certain unknown through elimination of others. Some of the exercise problems here will elaborate the idea more and I suggest strongly that at this stage the lessons given in chapter 3 and chapter 4 the corresponding exercises should be attempted to come at par with the lectures going. In the next lecture we will be discussing the Gauss elimination family of method to solve square system of equations which are most common in application and most important also. Thank you.