 Welcome back everyone to our lecture series on linear algebra based upon the textbook linear algebra done openly as Usual I'm your professor. Dr. Andrew misaligned. It's great to have you here today Today we're gonna talk about section 4.7 Which is entitled the Graham Smith algorithm now the Graham Smith algorithm are sometimes called in the Graham Smith process is a way of constructing an orthogonal basis The orthogonal projections and the like that we've talked about previously in the section like the foyer coefficients And what have you? typically dependent upon having an orthogonal basis it turns out that Using the idea of orthogonal projections. We can actually construct an orthogonal basis given any basis of a Subspace in fact it doesn't even have to be a basis if we have a spanning set we can construct an orthogonal basis using this Graham Smith process Then when we're done With construction orthogonal basis if we want an orthonormal basis we can normalize each of the vectors To make them unit vectors now some people present the Graham Smith algorithm as Normalizing along the way. I don't really see a need of doing that initially We can always normalize when we're done and the Graham Smith process it actually Constructs the ortho orthogonal basis in a recursive manner, and so let's explain what we mean by that So suppose we have already in hand a basis for our vector space w that sits inside of Fn and so what we're gonna do is recursively create a basis That's orthogonal in the following manner. Well, if that basis is empty then well, we're already done we don't need to do anything about that because the empty basis is Ortho orthogonal basis for the trivial subspace, but for a Positive dimensional vector space. We're gonna take we already have the basis x1 x2 up to xp And so we're not gonna do anything to to the first vector v1 Which would be the new basis that we're forming. We'll just take that to be x1 no change whatsoever But for v2 what we're gonna do is we're gonna take the current x2 and we're gonna subtract from it v1 dot x2 over v1 dot v1 times v1 that is we take the orthogonal projection of x2 onto v1 and we subtract it from x2 and this is gonna have the consequence that the new vector V1 will be orthogonal to the previous vector v1 So the current vector v2 will be orthogonal to the v1 using this orthogonal projection Projectionality and then for v3 we do the same thing. We're gonna take the current x3 We're gonna subtract from it the orthogonal projection of x3 onto v1 and the orthogonal projection of x3 Into onto v2 and when we do these together in tandem, this is the orthogonal projection Into the subspace spanned by v1 and v2 of the vector x3 and So this new vector v3 will be orthogonal to the vectors x To the vectors v1 and v3 v2 excuse me And so then we just replicate we repeat this process over and over and over again recursively v4 Will be x4 subtract the orthogonal projection of x4 onto v1 v2 v3 x5 Sorry v5 will just be x5 subtract the orthogonal projection of x5 onto v1 v2 v3 v4 and we keep on repeating this over and over and over again until we get to the last vector vp For which we take xp subtract from it the orthogonal projection of xp onto v1 The orthogonal projection of xp onto v2 onto v3 onto v4 all the way down to vp minus one. All right, and So We do this step by step by step and so each step along the way We're we're at we're replacing an x with a vector v which will be orthogonal to all the previous V's and we're gonna have the property that every time we replace a vector v Sorry, we replace a vector x with a vector v. We don't change the span So if we go from one up to K One up to K. We don't change the span of the original set x1 up to xk But we are replacing with orthogonal counterparts and the basic idea is the following x1 and v1 are the same vector So what interchanging them does nothing if you replace v2? That is if you replace x2 with v2 well notice that v2 is just x2 minus some multiple of v1 Since we have a v1 you could replace that away and you get back the original x2 and so when we come to v3 V3 is inside the span of x3 v1 and v2 which by mathematical induction We know we'll have the same span as the previous ones so V3 doesn't add anything new, but if we use v3 instead we could reconstruct We can reconstruct x3 using v3 in combinations of v1 and v2 so by induction This doesn't change the spanning set whatsoever So let's show you an example of how this process works in practice So take the following three vectors x1 which is 1 1 1 1 x2 which is 0 1 1 1 and x3 which is 0 0 1 1 and this is a this will form a subspace of R4 That is w is the span of x1 x2 x3 if we apply the Gram-Smith process to that what do we get? Well, the Gram-Smith process will reproduce the exact same basis if these vectors are already orthogonal But a quick check shows that they're not orthogonal If we take x1 and x2 together the dot product is 3 x1 and x3 gives us a dot product of 2 and Then x2 and x3 gives us a dot product of 2 again So these things are not orthogonal So step one of Gram-Smith is pretty easy. You just take v1 to equal x1 You don't change anything so just get 1 1 1 1 here make no changes whatsoever For v2 what we're going to do is we're going to take x2 and we subtract from it v1 dot x2 Over v1 dot v1 and the times that by x1 of v v1 excuse me So let's see the details of that. So the original x2 is 0 1 1 1 and So if we take the dot product of v1 and x2 x2 is the vector you see Just drawn right there and then v1 is still just x1 if we take the dot product of those We're going to get a 3 and if you take the dot product of v1 with itself you get a 4 and we're going to times that by 1 1 1 1 v1 there and So when we combine those together you're going to end up with 0 minus 3 4 so you get 3 4 right there You're going to get 1 minus 3 4 which is just a 1 4th and then that actually happens with all the other coefficients as well So you get 3 4th 1 4th 1 4th and 1 4th and that should be a negative 3 4th at the top there. Sorry about that And so that gives us a vector which is going to be orthogonal to the region 1 notice now if you take v1 dot This v2 you'll get negative 3 4ths plus a 4th plus a 4th plus a 4th that gives you 0 Now if you are completely important to all fractions The ratio phobic amongst us might be panicked here be aware that you could notice there's a common factor of 1 4th If you were to scale that out The 1 4th you get negative 3 1 1 1 Like so and this vector negative 3 1 1 1 is still orthogonal to v1 We're going to get still negative 3 plus 1 plus 1 plus 1 if we take the dot product And as scalar multiples don't change they don't change The span whatsoever. We actually could get rid of the 1 4th And just take v2 To be this vector right here negative 1 1 1 1 we're going to normalize all these vectors in the end So it doesn't really matter what scale the multiple you do if you prefer the integers That's perfectly fine. I do too I'm not afraid of them But you know, I I think integer arithmetic takes a little bit easier than working with fractions So we're going to use uh, we're going to use negative 3 1 1 1 for our v2 All right. So how do we do v3 then so we're going to take x3 Whoops, uh, we're going to take x3, which remember that was the vector 0 0 1 1 We're going to subtract from it v1 dot x3 So x3 is the vector. I just wrote down there v1 was all ones So you take the dot product you get 2 this will sit over 4 And you times that by 1 1 1 1 All right, and then we have to set track from this also v2 Which as we mentioned before v2 is now going to be negative 3 1 1 1 But we have to compute the foyer coefficients All right, uh, if you take v 2 dot x3 You end up with a 2 um, if you take V if you take v2 dot v2, you're going to get a 9 plus 1 plus 1 plus 1. That's a 12 sits in the bottom now this these numbers would be a little bit different if you had Used the original v1 v2 we had this one right here But the thing is we can scale this because these foyer coefficients will correct any scalar multiplication we have there That's why i'm perfectly happy using integers Uh, let's simplify some of the fractions because after all One half one one the the two fourths is the same thing as one half And the two twelfths we can make that become a one sixth And so adding everything together For the first component we are going to get zero minus one half Uh, we're then going to get plus three sixth And i'm going to help us out there Uh, let's see three six of course is one half zero minus a half plus a half is zero We like zeros Uh, for the second component we still get a zero we're going to get a negative one half And this time we're going to get a minus one sixth And so I would need to write the This one half here like you said we're afraid of fractions This should be three sixth So this will all add up together to be negative four sixth or negative two thirds And then the third component is going to be one minus a half Minus a sixth So this is going to take the negative two thirds we had before but we're adding one to it So we get a positive one third And then the last component is the exact same thing So, uh, we get another one third as well And so this is our vector v three if you don't like the fractions again, we could factor out the one third So we get zero negative two one and one And so then we could just take v three To be that scalar multiple zero negative two one and one like so And if you check with the previous vectors if you take If you take v three dot v one remember v one was the all ones vector So you're going to get zero negative two plus one plus one that's a zero And if you take v three dot v two which is up here negative three one one one What you're going to see happening there is you're going to get negative two plus one plus one So that's also orthogonal. So these three vectors are now orthogonal And so this is our new basis C We didn't change the first vector at all the all ones vector The second vector was then negative three one one one And then the last one we just got was zero negative two one and one And so this provides now an orthogonal basis an orthogonal basis for the original vector space w We did not change the vector space. We just changed That we now have an orthogonal basis for which then we can do all the cool things with an orthogonal basis For this vector space now, so we can always create an orthogonal basis If we need one Now this is what if we want an orthogonal basis and we also chose this so they're all integer numbers there if we want If we want an orthonormal basis Then remember the idea is we need to We need to normalize everything so we could take u one To be the normalization of v one So one one one one The length of that vector is going to be the square root of four So we get the vector of all one halves For the second one, we want to normalize that So we take the vector. What did we have before? negative three one one one And we actually calculated the length of this thing earlier remember Because when we were doing the foyer coefficients, we had to do v two dot v two, which gave us the 12 Well, now we just take the square root of that Which of course if you want us if you want you can rewrite the the square root of 12 as just Two root three I don't know if that's necessarily simpler. I I'm perfectly happy with the square root of 12 in the denominator there So you two we're going to get negative three over the square root of 12 one over the square root of 12 one over the square root of 12 And one over the square root of 12. So that's our second vector there And then for the third one just normalizing this thing U three The vector we had was zero negative two one one And so if we calculate its norm We're going to get four plus one plus one with just six square root of six in the bottom And again, just leave them. You don't need a rationally it's a denominator That's a silly thing that people tell us that they don't really give us a good reason negative two over the square root of six one over the square root of six And one of the square root of six So with these vectors in mind, we now have an orthonormal basis If we take if we take the set u one u two and u three This is now an orthonormal on orthonormal basis for w So if we have a basis for any subspace we can construct an orthogonal basis And then an orthonormal basis from that and like I said, uh, if If you just have a spanning set this process will automatically prune down that spanning set when you form the orthogonal basis It'll whenever you have a vector you didn't need It'll actually produce a zero and just throw it out of the process Now I want to do an example with complex matrices or complex vectors here Because one has to be careful when you do your foyer coefficients You have like the vi dot x j Over vi dot vi When you're working with complex inner products that is the Hermitian product The order does matter if you switch these two around on the top You'll actually get the conjugate and then you won't actually have an orthogonal basis that you need So do make sure you get the order correct As and so let's go through the let's go through the details of that right here So if we take our first vector u, we're just going to leave it alone We don't really need to modify whatsoever. So x1 the first vector we're going to construct right here We'll just be u 1 i and zero So to find the second one we're going to call this one. I'm not why we'll call it x2 I guess that makes more sense x2 is we're going to take the previous vector v And we're going to subtract from it u dot v over u dot u Right here and remember as we're taking the Hermitian product We take the A conjugate of the first factor So as we i'm going to be a little bit more detailed in the inner product here So if you take u dot v you're going to get one bar times one plus i bar times zero plus zero bar times negative i And this sits all above one bar times one plus i bar times i Plus zero bar times zero of course the bar is just the complex conjugate Times this by the one i and zero like so Now there's a lot of zeros here. So this does simplify simple enough one zero negative i And so the conjugate of a real number of course is just itself So you can get one plus zero plus zero for the numerator for the domain you get one plus One i bar times i is just going to be negative i times i which is one and then zero there zero And so that fraction make sure that looks like an i So the fraction we get negative one half times the vector one i and zero And so even though these are complex numbers, this will just be a linear combination like anything else We're going to get one minus a half, which is a half we get zero minus i halves, which would be minus i halves And then we're going to get negative i minus zero. So we get negative i right there And so this should be orthogonal to the original vector again If you don't like the fractions take out the one half That gives you one negative i and negative two i Like so and we can check and we're going to take our x two Because we don't we don't want the one half scalar. We're we're good without it And so we're going to take x two to be this vector one negative i and negative two i And so let's check to see in fact if we got an orthogonal set or not So as a reminder the first vector x one which we didn't change That was the vector one i zero And x two is what we just got a moment ago one negative i negative two i And so if we take the inner product of these things Remember we're taking the Hermitian product here x one dot x two We're going to take one bar times one We're going to get i bar times negative i And then we're going to get zero bar times negative two i Well zero times anything is easy enough. That's zero even for complex numbers You get one times one, which is a one and then you're going to get A negative i times a negative i that's a double negative. So you get one minus i squared I'm sorry i One plus i squared negative or i squared itself is negative once you get one minus one, which is zero So this in fact shows us that x one is orthogonal to x two Like we wanted to and this span didn't change We still have the same spanning we still have the same span that we did before so If we take the subspace span by those two vectors u and v x one x two will span the exact same thing So we have to be a little bit more careful with with um complex vectors, but other than that It's it works out pretty nicely And so what I want to then talk about here is To Talk about the the qr factorization of a matrix Which we've talked about various factorizations of matrices before particularly in chapter three We talked a lot about matrix factorizations like the lu factorization factorization using elementary matrices and the like The qr factorization offers yet another important factorization that's closely related to this Graham smith process that we've been talking about right now So imagine a is an m by n matrix with linearly independent columns It doesn't have to be a square matrix, but we do we do want that have independent columns If there is a dependence relationship on the columns this factorization isn't going to work Then if we have independent columns, we can factor the matrix a As a product of two matrices which we call q and r hence the qr factorization where q q is an m by n matrix So it'll have the same dimensions as the original matrix a but it'll be it'll have orthonormal columns. Whoops It'll have orthonormal columns to it which I want to make a comment that if If this is an n by n matrix, this is what we call q is an orthogonal matrix Which we talked about orthogonal matrices in a previous lecture 4.4, I think And orthogonal matrices have many ways of defining them They're those matrices whose transpose is equal to its inverse, but that's equivalent to having orthonormal columns Now again our matrix here does not necessarily have to have A square shape. It could be any rectangle Um, and so this is sort of like the rectangular equivalent of an orthon orthogonal matrix So q will be an or an orthonormal matrix and The column space of a will be identical to the column space of q So we're not changing the span of the column vectors here That's q uh r on the other hand is going to be An n by n so it will be square. It'll be upper triangular Matrix and all of the diagonal entries will be positive And in particular if you have a triangular matrix whose diagonals are non-zero That means it'll be an invertible matrix. So r will have an inverse to it And so I want to give you a quick explanation how one constructs this thing. So Uh, the proof is essentially the algorithm for one finds the q r factorization So you'll be given the matrix a you start off with that. So the first thing you're going to do Is you're going to Gram smith the columns of a Okay, and so then you're going to get this basis So the second step here is q is formed from This orthogonal basis this I should say or thought orthonormal basis that you just constructed here And maybe I should specify this in here in our description that we do want it to be orthonormal the orthonormal basis That is the columns uh the the columns The columns of q are going to come from this basis right here And so once q is in hand, it turns out r is pretty easy easy to compute. Um to compute r All you have to do is you take r equal to q transpose a And that's it Um now, of course, if you have a complex matrix instead of taking q transpose, you should take q star Um, because that's what's necessary to have the appropriate orthonormality of the columns right here And the basic reason this works is if you look at q transpose q Uh q transpose a what you're going to do is you're going to get a matrix all of which you have these inner products qi Dot ai where ai Where where qi is the columns of q and ai are the columns of a and so you take the transpose And work through multiplicate the matrix multiplication. You get a bunch of these dot products right here You get the all of these qi's times the ai's Like so and I should mention that um The order the order on which you place The original basis will kind of a will affect the gramm smith process Because the process is recursive if you change the order of the original basis that does change things a little bit And so in order to get uh the matrix r to have all positive entries, you might have to change the order of columns Uh that it's not such a big deal One can do it and make sure you get positive columns This will become actually more more obvious when we talk about the determinant In the next chapter because we do similar things when we change the order you change the sign of some things And so let's see an example of this. So I wanted to just use the same the same um Matrix we've been doing this whole time. So take the matrix you see right here Uh, it's a four by three matrix whose first column is one one one second columns is zero one one one and third column is zero zero one one The columns will probably look familiar. These were the x uh the x one two and three that we had earlier If we apply the gramm smith process to this You'll remember that column one became the all one half Vector zero one one one became negative three over root twelve one over root twelve one over root twelve one over root twelve And then zero zero one one became zero negative two over root six one root six one root six So if you apply the gramm smith process to these three column vectors you get these three Columns right here and as we saw before this is an orthonormal set of vectors So that's the hard part of the qr factorization. You apply the gramm smith process And so that's the little summary we'll have right here. You take this This matrix And you apply you apply the gramm smith process That's how you get this matrix q And then how do you get r if you take the formula we had before take q transpose a Now if q is an invertible if this if a is a square matrix q will be a will be a square matrix as well And you're going to get q transpose is just the same thing as q inverse And so that's basically what we're doing here once you have q and a you can solve for r by taking the by the inverse But as this is not necessarily a square matrix We can use transpose and get something that kind of mimics the inverse in general And so if you take q transpose, would you see right here? q transpose and this is the original matrix a if you go through all the possible products like one half one half one half one half Times one one one one that'll give you Four one halves which added to be four halves, which is two the first bit Uh, then the second one you're gonna take one half one half one half one half times zero one one one right You should get three halves in that situation And then the last one you'll get two halves, which is that and you know go through all the details This will give you your matrix r And we did position the vectors in such a way that we get all positive entries If this didn't work you might have to permute the order of these columns And then permute the order of these rows accordingly, and that'll actually give you that'll switch up the coefficients So that they're positive and this gives us our qr factorization In the next lecture we'll talk about an application of the qr factorization relevant to the least squares problem amongst other things But yeah to get the qr factorization really just apply the grand smith process and do a quick matrix multiplication after that I say quick, but I mean the the matrix q is can probably have some Square roots floating around here since it's an orthonormal set of vectors The columns are it's not going to be the prettiest Vectors you ever have but there are some huge benefits of having that orthogonal set and this Orthogonal factorization of the matrix A And so that concludes section four point seven Thanks for watching today. If you like this video, then please subscribe And just to help you know make some more videos in the future If you have any questions, please put them in the comments below if you like what you saw Please comment about those as well and I will see you next time as we conclude chapter four Thanks for listening. Bye