 The best possible approximation of a vector, upspace of the original vector. Us, we have a vector space which is also an inner product space. Within that vector space, we have this subspace sitting inside. The question is framed us for V sitting inside V ascertain whether some V hat inside U is best approximation of a lot of things to go here. What in terms here is the best approximation bit? What do we even mean by best approximation? Hope something being the best or something being better than something else. There has to be a metric. There has to be a measure. There has to be some attribute through which we qualify someone as better than someone else. Mathematics that has to be something numerical, quantified. So, what is it? What do you think is going to be this norm of as close to zero now how small? But is it possible to have it zero all the time? No. So, then how do we know that for a given choice of V hat that you have made, you are indeed sitting on the best possible choice and that no one can do any better. A matter of chance. You have tried something. So, we want a definitive way of establishing that the particular V hat that you have chosen from this upspace U is indeed the best you could have done and maybe no better. Or maybe at least if there are others, then at least you cannot do any better with any of the others. So far we are not making any claims for uniqueness. We will see later that that will fall. So, would you agree that if I formulate this problem in this fashion that V minus V hat is smaller than the norm of V minus U U more than or equal to for all U that is a comparison. So, if indeed I have hit upon the best possible approximation of V sitting then no matter what other U I pick out from the same subspace U form cannot be any less than this. So, in that sense it is the best. So, when we say best approximation then this is exactly what we need. Then V hat. So, if this is true then V hat is the so-called best summation of V in. This is how we should qualify something as a best approximation. Makes sense. Again you realize that this is like comparing two norms and it just defines what a best approximation is. But in no way does it give you any easier way to check that whatever V hat you have potentially picked out is indeed going to do better than any other U that your friend might pick out. You will have to go ahead and check against maybe infinitely possibly infinite number of vectors. What a test. Here is the important claim that we are going to make now. This is equivalent to or rather let us put it this way V minus V hat normally is not equal to V minus U for all U belonging to U is the same as saying that the inner product of V minus V hat is equal to 0 for all sitting inside U. The solution is as follows. Again comes from Euclidean space running after a balloon. Sources that the kid can get to the balloon is directly below it which is when the error vector. So, what is the vector joining the origin of your frame of reference to the balloon? That is the V. The kid cannot reach that vector V. The best that the kid can do is run along that earth which is V hat. And the best possible position that the kid can occupy is right below the balloon. In which case the difference between V the position of the balloon and V hat the position of the kid on the subspace that is the flat earth V hat that is perpendicular to flat earth. What does it mean? It is perpendicular to every vector that you draw on the flat earth which means it is perpendicular to every vector U. So, this error vector. This is the error in your approximation. This is the actual vector. This is your approximate the putative vector that is best possible approximation of V and U. Error vector is going to be orthogonal to every vector inside U. That is when we say that this is the best possible approximation. So, intuitively it all seems to fit in because our intuition is based on Euclidean spaces. Start or build up towards more abstract spaces. But now this requires a proof. I cannot just argue about it like you know as the way we do for Euclidean spaces we should give up more generic proof. And this is a both way implication. So, it means it is a necessary as well as sufficient condition. So, by checking for this you are not being conservative. If something fails to meet this condition then such a V hat will also fail to be the best possible approximation. So, this is exactly equivalent to improve this law. Suppose D minus V hat or V hat that you have chosen is indeed orthogonal to every you might choose from the subspace U. First we assume upfront that you have managed to find a V hat such that the difference between V and V hat is a vector that is orthogonal to every vector in the subspace U. Now what we have to show is this. The difference between V and V hat and its norm and that is going to be the least possible norm among all possible other choices that you might make from U as a potential candidate for a best approximation. We have to show this part now. We have assumed this we have to show this. So, let us see. So, V minus U the norm squared is nothing but V minus V hat plus V hat minus U. What is this equal to? How have we defined the norm in an inner product space? This is nothing but V minus V hat plus V hat minus U inner product with V minus V hat plus V hat minus U. Observe, what can you say about this vector? This is brackets. So, on that I have put an under brace on. Where does this come from? It comes from U right because V hat is your claimed best approximation and U is some arbitrary vector. Both are coming from U. Of course, they are coming from V. That is not relevant. The bigger picture is that this is coming from U. So, this belongs to U. This belongs to U. So, if I now open up this bracket or rather if I take this as term one, term two, term one, term two and use the linearity or the homogeneity part here. So, it is just the additivity. So, now because we have chosen this from U, what can we say? Let us write it term by term. So, we have V minus V hat inner product with V minus V hat plus V minus V hat inner product with V hat minus U. Plus V hat minus U inner product with V minus V hat plus inner product V hat minus U with itself. Now, among these what can we say about this term and this term? Look at our claim. We have claimed the other way round that is V minus V hat is orthogonal to every vector in U. So, V minus V hat should it not be orthogonal to V hat minus U? Because V hat minus U as we have seen also comes from U, right? Any doubts about this? So, this obviously goes to 0 as does this. So, what are we left with then? This is nothing but the norm of V minus V hat squared, is it not? So, this is norm of V minus V hat squared plus what is this? Norm of V hat minus U squared. So, for any arbitrary vector U that I have picked out from here, yeah? It turns out the norm of that squared is equal to the norm of this fellow squared plus some positive number. So, this is definitely greater than or equal to norm of V minus V hat squared, but that is exactly what I was required to show, right? You can just take the square root now. So, maybe I will run out of colors at this rate. So, this quantity is greater than or equal to this quantity and equality can only hold if this vanishes. But if this vanishes, then you have made no different choice of V hat than, I mean, no different choice of U than V hat itself, which defeats the purpose of saying that, you know, there are others who can do better, right? So, therefore, when you have this condition satisfied, it means that you have hit upon some V hat which is the best possible approximation of V within the subspace U. But we have to show the other side of the proof as well, which is that if we claim, so this might make it appear like this is a sufficient condition. What if you don't meet this condition? Is it still possible for you to, you know, find out or claim that such a vector which does not meet this condition? Maybe that is also a best possible approximation. This is only one side of the proof, right? Because we have assumed this to be true and we have shown that that implies this. But if you have to complete the proof, we have to show that this inequality, if we start with this blue part, if we start with this blue box inequality, then that must lead us to logically conclude that this must be true. So, that we haven't yet shown, which is what we are going to try and show now. So, suppose V minus V hat is less than or equal to V minus U for all U in U for some V hat coming from U, right? Once again, let us consider V minus U square of the norm and you will feel like this is deja vu because we are doing the same thing again. V minus V hat plus V hat minus U whole squared. But now we cannot assume both sides, then there is no point in the proof. We cannot assume that V hat minus U is going to be orthogonal to V minus V hat because that is what we have to show. We cannot assume that now. So, we have to retain all possible terms now. So, again, this is going to be the same as before. So, maybe I can skip this. You can already see that this is going to lead to this, right? So, the next step would be this. But I would not be able to use this pink part that I have used here. So, I am going to write the next step following the pink part here. You can fill out these two lines, yeah? So, what does it mean? This basically means, does it not that V minus V hat squared, the norm squared plus V hat minus U the norm squared. What about this? What can you say about this? See, this is just the order flipped. So, can I not write this as the sum of a complex number and its conjugate? Agreed? This is V minus V hat inner product with V hat minus U. This is V hat minus U inner product with V minus V hat. So, I can just flip this order and say that this is the complex conjugate of V minus V hat with V hat minus U by flipping the first and the second argument in the inner product. Nothing more than that, right? So, in that case this is just 2 times the real part of the inner product that is this. So, I am going to write that as plus 2 times the real part of the inner product of V minus V hat with V hat minus U. Earlier, when I had assumed upfront that this property is true, this automatically vanished. So, that is what the proof was, but now I have to worry about this term. I cannot assume, I have to conclude as a result of my proof that this is 0, yeah? Okay, but notice I have indeed assumed at least something which is this. What does that tell me? If I now write this V minus U the norm squared minus V minus V hat the norm squared that is equal to the norm of V hat minus U the norm squared plus twice the real part of this inner product V minus V hat and V hat minus U. What can I say about this quantity, this object? By my supposition there, whatever I have supposed, what can I say about this? What should this be? Sorry? Greater than or equal to 0. Greater than or equal to 0, is it not? Because I have assumed now that this V hat is the best possible approximation. So, any other U that you have picked out you cannot do better than this V hat. So, this part of this must be greater than or equal to 0, sorry, greater than or equal to 0, which means I can write that this V hat minus U the norm squared plus twice the real part of the inner product of V minus V hat with V hat minus U is greater than or equal to 0. Must be true for all U, right? Because U is allowed to be arbitrary. I am searching over all possible U's that I can pick out in the big U that is the subspace of V. So, this is for all U coming from U, right? So, I am going to erase this part now and continue here. Remember what I am free to choose is this U and that is exactly what I am going to exploit to contradict to arrive at a contradiction. I am allowed to choose any vector U from the subspace U, right? So, consider U is equal to V hat plus inner product of V minus V hat with some arbitrary U bar divided by norm of U bar squared times U bar for some arbitrary U bar in U. Is this a legitimate choice of U? What does U need to satisfy? U needs to belong to the subspace U. Look at how I have proposed that you choose this U. V hat obviously is part of U. U bar is some arbitrary U, a vector in U and it is just a linear combination of those two. So, that must also definitely, I mean the fact that this belongs to U is true. So, this is a legitimate choice of U. Now, with this choice of U, I should satisfy this inequality. Because this inequality must hold true regardless of what U I choose. I am allowed to sweep over all possible vectors in subspace U, right? So, this U that I have chosen now must also satisfy that inequality. If it does not, then I would have contradicted, yeah? Then I would have contradicted whatever I am trying to establish here, right? And that contradiction must lead me to somehow conclude that this V minus V hat should be orthogonal to every vector in the subspace U. So, we will see how that is, that follows. Let us look at this first term. When we plug in this U, what happens? V minus, sorry, V hat minus U, the norm squared is what? Exactly. So, V hat minus, V hat minus this. So, you are left with norm of minus V minus V hat with U bar upon the norm of U bar squared and then U bar. So, remember this is just a scalar. This is just a scalar like some alpha and this is the vector. So, what is this equal to? Isn't this equal to V minus V hat inner product with U upon U squared whole squared, sorry, U bar whole squared times the norm of U bar squared agreed? Check. So, what I am doing is the following. I have chosen a, I have shown that this must be true for every possible U inside the subspace U. And then I have made this choice and this is the winner. I made a choice of one such U in U and if this is to follow, then I am checking term by term what these turn out to be and I am going to add them and check if this inequality still holds. So, in the process of doing that, I am just checking out what the first term becomes. Right? And I am claiming that this is what the first term becomes. But of course, this can be simplified. So, this is nothing but norm of V minus V hat with U bar the inner product squared upon norm U bar squared just a cancellation of this. Yeah? Of course, I am not choosing a 0 U U bar. That is trivial. Right? I am not choosing U to be just V hat. So, this division is legitimate. Right? So, this is what it is. Let us look at the second term. Rather, let us look at just this term, this inner product. And we will see that whether you look at the real part or not would not matter. Okay? Let us see. Consider V minus V hat with V hat minus U the inner product. Okay? So, V minus V hat remains as it is that is nothing but V minus V hat. What happens to this? V hat minus U we already know what it is. It is exactly this term. Is it not? Yeah? So, it is an inner product with the scaled version of this. Right? So, it is minus inner product of V minus V hat with U bar upon norm of U bar squared times U bar the inner product thereof. Right? What does this lead to? Look, this is a scaling factor in the second argument. So, it has to be conjugated because of the sesquilinearity. We cannot pull this out without conjugation because it is a scaling factor of the second argument, not the first argument. If it was the first argument, we could have just pulled it out as it is. So, what is this going to turn out to be? This is going to turn out to be minus V minus V hat inner product with U the conjugate thereof upon U bar squared and then what about this? It is nothing but the inner product of V minus V hat with U bar. So, if I multiply by 2, which is what I will eventually be required to do, it is just 2 and 2 here. So, what is this? This is the product of a number with its conjugate, product of a complex number with its conjugate. What is arbitrary here? The choice of U bar, some nonzero U, some nonzero vector in the subspace U. Right? What does this turn out? Does it make any difference whether you take the real part of this or not? It is already real, right? It is the product of a number with its conjugate is just the squared of the moduli. That is a real number, yeah. So, what is this? This is nothing but 2 times, you will have to excuse this small writing here. Maybe I will erase this part here. This is 2 times the inner product of V minus V hat inner product with U bar squared upon the norm of U bar squared with of course, the most critical bit this minus sign coming ahead of it. So, what happens if you add this with this? See, first observation whether you apply the real part of this, whether you filter out the real part of this or not doesn't matter, it is real anyway. So, you can go ahead and put a real ahead of it just to satisfy yourself, but it won't make any difference. So, if you take the real part of this, you are still going to end up with this. Now, if you add this and this, what do you end up with? Shall I write that down there? So, therefore, I will just keep this boxed term here and I am going to write the following. Our choice of U in U, we have norm of V hat minus U whole squared plus twice real part of V minus V hat inner product with V hat minus U upon nothing upon. So, this is inner product and put that angled bracket. What is this equal to? Minus, see there is plus 1 and there is minus 2. So, you end up with minus the module I of the inner product that is V minus V hat with U bar squared. So, that is definitely a positive number and the denominator is the norm of U bar squared. So, what do you, what can you say about this number? This is less than or equal to 0. So, you arrive at a contradiction unless between these two you are sandwiched. So, that you have to conclude that this is equal to 0, right. So, let us put that in a blue box here. So, based on these two, what am I supposed to infer? What I am going to infer from those two is that V minus V hat inner product with U bar is equal to 0 for arbitrary U bar in U which is exactly what I was required to show, right. Please pay very careful attention to the second part of the proof. The first part we finished off pretty quickly and it was quite straightforward. The second part required a bit of work. We had to come up with this fact that this is greater than or equal to 0 and then we did come up with the choice of U which makes it less than or equal to 0. The both cannot simultaneously be satisfied unless it is exactly equal to 0 and if it is exactly equal to 0 then we are led to concluding that for arbitrary choices of U bar inside the subspace U you must have the vector V minus V hat the error vector to be orthogonal to it. Since U bar is arbitrary it means you could have chosen any vector in the subspace U and this orthogonality would have held which is what the premise was to start with, right. Any doubts on this? I wanted to come at a come up with a contradiction. I wanted to prove certain things. So, if I can massage this I mean if you ask me logic something suggests to me that somehow I have to contradict this. So, if you are asking me that is your what you are asking is about the method. So, this is not madness there is a method behind it. So, what is the method? I am already seeing something interesting here this part is going to be always positive. The only way that I can overturn this condition is if this part somehow messes things up and spits out a negative real number. So, I have to somehow choose some U so that it spits out a negative real number sufficiently negative so as to offset this and make this whole thing negative definite negative semi definite at least right that this inequality can be held and I have checked that this one does the work. I mean in the proof it is not a trial and error this does work as you have already seen but yeah if you are asking me a more philosophical question like when you are encountered with such problems in an exam or in an assignment and how do you solve it then you have to definitely try I mean this is not going to be very obvious or apparent by any stretch of imagination you have to try out several things and that is how you develop intuition you solve more and more problems and you get a hang of what might and might not work but I am not certainly going to expect that if this proof were asked in an exam or in an assignment you will get it at the first shot there is no trick there is no obvious way to see why this should work. Yeah you can interpret this post facto what I am saying is now that you have this before you you can definitely interpret this post facto that you want something that is in the opposite direction. Yeah exactly it is along some arbitrary direction that you choose on to the on to some arbitrary vector in the subspace u yeah exactly and you are going to add that to v hat yeah. So, you are basically saying that if this is v hat let me go ahead and add some arbitrary perturbation to this v hat in some arbitrary direction that is determined by u bar that is what this is doing. So, you have a point you start go up to this point v hat and then you sort of take any arbitrary direction. So, this is the direction of u bar and then you are basically allowed to choose any vector in the subspace u by that sort of a choice of u bar and then you are checking against the error vector for each of these different choices that you make and what you have shown is that for any other choice the slightest amount of deviation even from u bar will land you up with a more greater penalty in terms of this. So, that this v minus v hat being orthogonal to the any vector in the subspace u is both the necessary and sufficient condition for the best approximation.