 So, in the next module now we shall see what is the benefit of having such an orthogonal set particularly in view of the problem we have concerned ourselves with for such a long time that is to solve equations like A x is equal to B alright. Now consider A matrix that is given like so A 1, A 2 until A k alright. So, what are these? These are all tuples of numbers ok. So, suppose these are all m tuples suppose this is the case and you have been asked to solve as usual A x is equal to B right. And suppose you have been told so exactly the setup for what we have just studied suppose that the columns which are familiar vectors like tuples of numbers m tuples of numbers. So, these columns of numbers that we have now A 1 through A k these constitute an orthogonal set of vectors in the Euclidean space m-dimensional Euclidean space and they do not contain 0. So, immediately what do you know? Full column rank obviously right because this is a linearly independent set. What sort of a matrix do you think this is if it is full column rank? Can it be a fat matrix? Tall, square at best, but in general it is going to be tall. So, obviously we will assume that m is greater than or equal to k right. Now with this additional condition that is being given to us we shall now see how this translates to a simple solution ok. Any questions? So, what we have is basically something like this x 1 times A 1 plus x 2 times A 2 plus dot dot dot till x k times A k is equal to B and we want to solve for these x 1 through x k where of course x is equal to x 1 x 2. So, as this is this is the column picture of the system of linear equations instead of looking at them as equations you know those hyper planes and those biases that we have introduced this is just those columns and does this does there exist a linear combination of the columns of A that equals B. Conversely the question is does this vector B which is already a fixed vector given to us belong to the image or the column span of the matrix A yeah and if so then find me that particular combination. So, now that we have been given that this is an orthogonal set what do you think we should go ahead and do? So, orthogonality comes only in case of inner products. So, obviously there is an inner product in this space and you know these inner products these are just dot products real vector spaces is also familiar to us. So, what do we go ahead and do? What do you think we should do? So, this is my important equation let us name this equation 1 take the dot product of both the sides with any A i yeah. So, we can take the dot product because this is a scalar. So, the dot product will be essentially with this vector yeah and then every time we take the dot product with any A i its dot product with or inner product with the other A i is apart from the chosen i itself will vanish because of the orthogonality. So, what I can say from here is that x i is going to be nothing, but the inner product of A i with B divided by the norm of A i squared do you agree? So, this is exactly that scalar that I am looking for and what will it look like in terms of matrices then? What is what operation is this equivalent to? If I want to write it in a compact form? So, suppose I stack these fellows up like so A 1 transposed A 2 transposed A k transposed is this not the same as A transposed yeah. So, if I hit this equation here maybe use a different color. So, if I hit this purple equation with A transposed on both sides what will that lead to? A transposed A x is equal to A transposed B right. What do I know about the structure of this A transposed A? Now, take a look at this A transposed and take a look at this A and tell me what happens when you get this multiplication done like A transposed A identity? Not really identity. Yeah, it is a square matrix of course of size k cross k no doubt it is just a diagonal matrix or any of the diagonals 0 can any of the diagonals be 0? Why not? Because we have ruled out this. So, what you are essentially looking at then let me erase this starred assumption which is all too important here, but what we then land up with is something like maybe I should write the second term too. So, maybe I can also write this A transposed B as what is this going to be and there you have it exactly this expression is it not because each of those diagonal entries yeah they are non-zero. So, this matrix is invertible this is a diagonal matrix as the D inverse exists right. So, this is the motivation behind looking for this property of orthogonality, but we are not done yet I mean if life was so easy that every time you encounter an equation of the form A x is equal to B but the columns of A are orthogonal conveniently then all you need to do is just hit them with A transposed on both sides and just invert a diagonal matrix which is very easy to do and that is it yeah. So, life is not always that simple. However, we might come up with a system of equations like so where except for the orthogonality there is something nice by that I mean that at least this is full column rank. Now, we wonder then that if we are given a system of equations of the form A x is equal to B with A having full column rank which is to say that the columns of A are linearly independent, but no claims about orthogonality can we somehow massage this yeah to look at an equivalent system of equations which does indeed turn out to be like so with that assumption going that is the columns are not just linearly independent, but they are also orthogonal and do not contain a 0 vector. So, given that then the more sophisticated question is when you bring yourself out of this realm of matrices is given a bunch of vectors that are linearly independent can you cook up another bunch of vectors that are not just linearly independent, but also orthogonal and on top of that what more you must be talking about the same problem right. So, what is sacrosanctin this problem exactly? So, the image of that matrix A that is given to you whose columns may not be orthogonal and the image of the new matrix that you might be getting should be identical because you are looking for a solution to the same problem. So, you should want this new set of orthogonal vectors which you might have cooked up if possible to also span the same vector space that is a span of this set of linearly independent vectors that has been given to you through columns of an arbitrary matrix that is going to have full column rank must be the same as the span of the set of the same number of orthogonal vectors ok. So, the question is can this be done can we get such a set of orthogonal vectors and the beauty about this result is not only is it an affirmative answer that is it can be done, but it is a very constructive answer. The way the proof of this result follows also shows you not just that fact that such a set exists, but also how to go about constructing such a set such an orthogonal set. So, that is what we shall call the we call rather the Gram-Schmidt procedure and we shall now take a look at how this allows us to go from any linearly independent set to a an orthogonal set of vectors which span exactly the same. In fact, it does a little more than that as will be apparent from what we shall now show, but I want you to remember that the motivation is to again solve problems like these Ax is equal to b right the motivation stems from there. So, here goes suppose v an inner product space has a set of independent vectors say s is equal to v 1 v 2 till v n there exists a w given by w 1 w 2 w n such inner product of w i with w j is equal to 0 for i not equal to j that is the first which essentially means that w is an orthogonal set of vectors right. The span of v 1 v 2 v k is equal to the span of w 1 w 2 till w k for k belonging to the set 1 2 ok something very very interesting. So, we are not just saying that the overall set w is going to span the same subspace as the span of s, but we are also saying that if you go ahead and pick out some arbitrary number of fellows from this set s ok and you go ahead and pick out the same number of fellows starting from the left that is the order must be preserved then each such subspace within that subspace overall span must also equal ok. So, of course, it also comes with another benefit that if i is equal to j you can make all of these unit norm in which case sometimes people use the term orthonormal set. So, in the inner product of a vector with itself is which is a norm of the vector is unity it is a unit norm vector. So, when you have a set of vectors each of whose norm is unity and each and it is an orthonormal orthogonal set you sometimes say it is an orthonormal set of vectors ok. So, this is basically the claim that is there in this Gram-Schmidt procedure ok. So, you have corresponding to this linearly independent set you have this orthogonal set right and once you have that then you can reformulate your problem of solving A x is equal to b as is obvious right there is a column span of this and the column span of this would be equal right. So, it is just about a choice of basis if you can represent a vector b in terms of the basis that is given by the w's you can also represent it in terms of the basis given by the v's you know how the basis transformation works right. So, I want you to understand that once you have solved it for this it is as good as solved for this too right. So, that is a huge plus when you are dealing with inner product spaces. So, how do we go about proving this the proof will constitute in showing the following facts. So, I hope I can erase this right this is clear. So, we shall proceed by mathematical induction ok. So, by mathematical induction that is going to be our go to strategy suppose n is equal to 1 I mean there is really nothing to prove in that case s is just v 1 choose w to be just v 1 upon non v 1 and you are done nothing to prove here really right. So, this is the base step suppose now n is equal to 2 what do you do any ideas what can we do now suppose you have the set v 1 v 2 once again you can go ahead and choose sorry this is w 1. So, again you can go ahead and choose w 1 in much the same manner as you chose it here, but how do you choose w 2 projection we have not yet spoken about what projections are we will deal with them shortly, but not in today's lecture surely. So, what do we do what is what do you mean by projection basically maybe in terms of the definitions that we have introduced so far you can talk about it again dot product is very specific to Euclidean spaces. So, you can actually dictate the term to me if you what do you think this is going to be. So, for n is equal to 2 s is equal to v 1 v 2 yeah. So, let w 1 is equal to v 1 by non v 1 oh I do not need this right. So, this is sorry yeah that is what it is what about w 2 what can I do about w 2 how do we define w 2. So, this is remember, but what we are defining. So, you might as well put this symbol here what do we do yes v 1 minus or rather v 2 minus what do we take what do we subtract from it that is it that is just a scalar right sure something is missing. So, what we are saying essentially is v 2 minus v 1 inner product with v 2 times v 1 upon norm v 1 squared. So, of course, you can go ahead and check that the span of w 1 is the same as a span of v 1 there is only two things to check here and then you have to check that the span of w 1 v 1 v 2 is equal to the span of w 1 w 2. So, first of all just notice here what is given to us that v 1 v 2 is a linearly independent set. So, can this w 2 vanish I wonder because if this is w 2 vanishes then it is no good to us can w 2 equal to 0 why not if w 2 is equal to 0 then that means, this equals 0 if this equals 0 then this is a linear combination of linearly independent fellows with a certifiable non-zero coefficient at least here. So, therefore, this linear combination cannot vanish because v 1 and v 2 at least is a linearly independent set that is given to us. So, therefore, this w 2 is definitely a non-zero non-trivial vector what is the next thing we have to check then this thing that we have constructed is it indeed and part of an orthogonal set that is inner product of w 1 and w 2 going to be orthogonal at all of course, you might say we might still need to normalize this, but it does not matter you see do not normalize this do not normalize this let us not choose an orthonormal set let us just choose an orthogonal set that normalization we can do when we are actually constructing explicitly those vectors for the time being we have just claimed orthogonality not orthonormality. So, let us just take the inner product of w 1 and w 2 what happens that is like taking the inner product of v 1 with this. So, this is equal to so, let me draw a line here is equal to v 1 v 2 inner product minus v 1 inner product with itself which just cancels out right actually I think we should have sorry I think we should have flipped this no yeah this should have been flipped it is basically v 2 minus v 2 inner product with v 1 not v 1 inner product with v 2 yeah because that is what works for us yeah because the order matters you see it is a it can be a complex inner product space as well in which case we cannot be frivolous with the order of this inner product. So, then of course this is 2 this is 1 yeah. So, now you see this is just v 1 inner with v 2 and this when it gets pulled out it is the complex conjugate. So, v 2 v 1 complex conjugate is v 1 v 2. So, this is v 1 v 2 upon v 1 norm squared and this is also v 1 norm squared. So, of course these get cancelled out this is equal to 0. So, indeed it is orthogonal yeah what mode is there to be shown that the span of these 2 fellows must be the same as a span of v 1 and v 2, but if you look at w 2 here what can you say from this expression if you look at this expression here this immediately leads us to conclude that w 2 belongs to the span of v 1 v 2 that is the very definition of something belonging to the span of a bunch of vectors it is written as a linear combination of v 1 and v 2. So, it belongs to the span of v 1 and v 2 now if it belongs to the span of v 1 v 2 then what can we say this is contained inside span of v 1 plus span of v 2 I am not claiming anything about direct sum or anything right it is true is not it. It is a sum that containment is already known right, but what is this yeah. So, what can we say about w 1 now w 1 is belonging to the span of v 1 and w 2 belongs to the span of this. So, what can we say about the span of w 1 and w 2 it is contained inside the span of v 1 and v 2. So, span of w 1 w 2 is contained inside the span of v 1 v 2, but here is the deal what is the dimension of this vector space what is the dimension of this vector space. So, you have a two dimensional vector space contained inside another two dimensional vector space what is the obvious conclusion. So, this dimension is equal to 2 this dimension is equal to 2 see that is the other side of the proof right we have said if you want to show two vector spaces to be the same you show both sided containment or if you can show one sided containment and the fact that the dimensions of those two fellows are the same then you do not have to show both sided containment instead you can just argue that they are one and the same right. So, based on this we have the span of w 1 w 2 is the same as the span of v 1 v 2 is that clear. So, at least for the base steps we have seen right n is equal to 1 n is equal to 2 we have seen this. The reason why I did this in somewhat detail is because now when we extend this for the inductive step we have to assume that this is true for some p for n is equal to p and then we have to extend it for n is equal to p plus 1 and show that given it is true for n is equal to p it must be true for n is equal to p plus 1 and you will see how the arguments and the reasoning and everything follows exactly these steps that is why I did this step for 2 normally people would do just this step for 1 and then go for the inductive step, but it helps because this is already a smaller vector space with only 2 members. So, you do not get addled with too many variables here, but now what we are going to do is go for the inductive step. So, this much is clear right n is equal to 1 n is equal to 2 the fact that Gram-Schmidt procedure works and gives us an orthogonal set corresponding to a linearly independent set is at least proved or shown this. This is just you know simple thing. So, this is in the span of v 1 and v 2. So, this is the span of v 1 plus v 2 and this I mean this is not that important here, but the fact that this is contained inside this and w 1 is also contained inside span of v 1. So, that is basically contained inside one of these fellows. So, the total span of these 2 fellows because this fellow is also coming from this alone. So, it is also contained inside this and this fellow also contained inside this. So, yeah. So, therefore, this entire span is contained inside this entire span and then we know that this is linearly independent. This is also linearly independent because this contains 2 non-zero vectors first is non-zero because v 1 is non-zero second is non-zero as we have explicitly shown. So, 2 non-zero orthogonal vectors must span a 2 dimensional subspace. So, therefore, this is of dimension 2, this is of dimension 2 and therefore, they are equal right ok. So, now consider v 1, v 2, v p to be linearly independent with. So, this is s with w is equal to w 1, w 2, w p orthogonal such that the span of v 1, v 2, v k is the same as the span of w 1, w 2, w k right. So, that is going to be our inductive step the first thing. So, this is assumed to be true for p.