 Let me recall we are probing this result let W be a finite dimensional subspace of an inner product space V let E be the orthogonal projection orthogonal projection of V on W okay then E is an idempotent linear transformation E is an idempotent linear transformation the idempotent linear transformation of V on to W on to W so we must show that this is an on to a map such that the following hold up first condition is that W perpendicular is the null space of V and so remember in order to write this we must know that E is linear see what we know is that the null space or range space of a linear transformation is a subspace so in order to write this we must know that E is linear okay this is one condition that E must satisfy and more importantly V is the orthogonal direct sum of these two subspaces okay for the proof I remember that we have done E is idempotent and E is linear okay so let me just write this already we have shown that E square equal to E that is E is idempotent and E is linear we must show that V is on to then W perpendicular is null space of V and this okay let us first dispose this off this is straight forward V is on to W that is given any X and W can I find okay given any U and W can I find an X in V such that EX equals U it is a same thing if so let me write like this if X belongs to W then what we know is that EX equal to X so the pre-image is X itself so that E is on to let me just emphasize W see E is a linear operator on V so from V to V it is not an on to map but if you look at the subspace W think of E as a mapping from V to W then it is on to okay that is what this means so it is the orthogonal projection of V on to W we must show that W perpendicular is null space of E okay so let us take we need to show this and then this let us take okay let us say Y or U or Z okay Z belongs null space of V implies E Z is 0 by definition in fact if and only if is not it by definition but E Z equal to 0 if and only if remember that to any fixed vector Z in V E assigns the best approximation the unique best approximation to Z from W so and we also know that condition what is that condition if EX equal to U then X minus U is perpendicular to W this happens if and only if X minus U is perpendicular to W that is I am writing it like this Z minus 0 belongs to W perpendicular that is the same as saying Z belongs to W perpendicular and so we have proved in one single step that the null space of E is W perpendicular okay okay this is just the definition how would how is E defined to any vector Z it assigns the unique best approximation and that best approximation U satisfies X minus U perpendicular to W that is what I have used okay finally this equation we need to show that but that is straight forward we show that V is W direct sum W perpendicular let us take X in V then X can be written as I want it to be in this plus this so I can write EX plus X minus EX let us call this as X 1 and this as X 2 X 1 plus X 2 then X 1 belongs to range of E but range of V is by definition W and X 2 is X minus EX so I want you to look at EX 2 EX 2 is E of X minus EX that is 0 EX minus E square X but E square is E so EX 2 is 0 just now we have shown that null space of E is W perpendicular so X 2 belongs to W perpendicular so for one thing we have shown that V can be written as a sum of W 1 sorry W and W perpendicular we must show that it is a direct sum that is we must show that the intersection is the subspace consisting of the 0 vector alone but that is straight forward. So what we have shown is that V is contained in W direct sum W perpendicular so I can write equality because this is a subspace direct sum is also a subspace so this is equal I am sorry W equals just W plus okay finally if let us say Z belongs to W intersection W perpendicular then by definition the inner product of Z with itself must be 0 because Z is orthogonal to every vector in W and Z belongs to W perpendicular also but this means Z is 0 so the intersection is singleton and so it follows that V is the direct sum of these two subspaces. So remember that there is no restriction on V the restriction is only on W it must be a finite dimensional subspace V could be an infinite dimensional inner product space but if W is not a finite dimensional subspace then this is not true okay if time permits I will provide an example later okay so this is a finite dimensional result okay corresponding to E we have this result the version for F which kind of complements E that version is similar so let me give you that version for the record. So the following is a consequence of the previous result remember the mapping F, F is I minus Z okay and we have seen that F is a projection of V onto of V on W perpendicular but what follows from this theorem is that it is an idempotent linear transformation of V onto W perpendicular this time and what is the null space of F what do you expect the null space of F to be what is F? F is I minus E null space of F is W perpendicular I am sorry just W okay null space of F is range of I minus E okay so let us null space of F is range of E yes that is W so let us quickly dispose this one corresponding to E we defined an F so let W be a finite dimensional subspace of an inner product space V and E denote the orthogonal projection of V on W as before then I minus E is an idempotent linear transformation I minus E is an idempotent linear transformation of V onto W perpendicular such that the null space of I minus E is W so previously we had W perpendicular to be the null space of E that is I am looking at the null space of the orthogonal projection at any stage the null space of I minus E is W the proof is almost there so I will not write down the fact that E is idempotent and linear implies that I minus E is linear in the first place and I minus E we have seen this last time I minus E square is I minus E so it is an idempotent linear transformation I minus E that is the mapping that takes X to X minus E X we have seen last time that it is it is the orthogonal projection of V on W perpendicular that was proved we called it F the mapping F from V to V defined by F of X equals X minus E X is the orthogonal projection of V on W perpendicular but from what we have seen just now it must be onto also similar to E I minus E behaves very similar to E but it is complementary to E so it is an orthogonal projection of V onto W perpendicular so this is done and null space of I minus E equals W is straight forward very similar to what we have done before okay so I am going to leave this last part also okay before so this is kind of the summary of the notion of best approximation how it helps in decomposing an inner product space in terms of a finite dimensional subspace and its orthogonal complement okay so what is important is the notion of best approximation okay but remember that for finite dimensional subspaces the best approximation exists and it is unique that is I mean these two are important for us to do all these things using again this notion we will go to the concept of adjoint of a linear transformation then the notion of unitary operators normal operators finally spectral finite dimensional spectral theorem okay but before that let us look at two examples two numerical examples where I will show how okay where I will try to give you another clue as to what you must expect of the linear transformation E when it is the orthogonal projection okay I asked to this question what is the difference between an ordinary decomposition direct sum decomposition and an orthogonal direct sum decomposition okay you may be able to answer by looking at this example so this is kind of reinforcing what we have done till now. So let us look at this example really two examples let us take R2 for R2 I have the following two decompositions let me call W1 as a subspace that is span of 1, 1 W1 is span of 1, 1 W2 is span of 1, minus 1 and I will write these are row vectors these are row vectors then what is the relationship between these two can I say W2 is W1 perpendicular okay observe that W2 is W1 perpendicular let us look at another pair I will call it Z1 Z1 is span of I will take the same vector as before so Z1 is really W1 Z2 let us say is span of the vector 1, 2 span of vector 1, 2 in the first instance obviously W1 direct sum W perpendicular is R2 in the second case can I say Z1 direct sum Z2 is R2 first answer must be instant answer is yes why these are independent vectors together they form a basis and this vector does not belong to the span of this this does not belong to the span of this so the intersection is single term 0 okay so for the record R2 can be written as W1 direct sum W2 perpendicular and equal to okay let me write like this and R2 equals Z1 direct sum Z2 the intersection is single term 0 and these two vectors together form a basis so any vector can be written as a linear combination of these two so for sum Z1 plus Z2 is R2 is clear Z1 intersection Z2 single term 0 is also clear because these are independent vectors just by inspection W1 direct sum W2 or I will write W1 perpendicular yes okay let us find the maps E in both these cases okay what is the property of the map E that we are going to use the subspaces are given we want to find E what I will do is not find E I will just take E to be a matrix itself remember there is this 1 to 1 correspondence between the matrix of a linear transformation and the transformation itself I will write down the matrix of E relative to the standard basis and see how it looks like by the way can you geometrically differentiate between these two direct sum decompositions? Geometry for the first one it is 1, 1 you can think of the line Y equals X okay that is the line which makes 45 degrees with the positive X axis the line 1 minus 1 okay makes 135 degrees with the positive real axis and observe these two lines are perpendicular these two lines are perpendicular in the second case this is the same subspace but look at Z2 Z2 is not perpendicular the line passing through 1, 2 is not perpendicular to the line passing through 1, 1 okay but then you know that since they are independent the span of these two vectors must be the entire space but remember that the angle between these two lines one passing through 1, 1 the other one passing through 1, 2 are not perpendicular this is crucial in the first case in the first case I want to determine E such that okay let E be the matrix I am using the transformation notation itself to denote the matrix let E be the matrix corresponding to okay the projection of W1 onto W2 and so E is let us say alpha, beta, gamma, delta E must be a 2 by 2 matrix okay it is a linear transformation on R2 E is this to determine E completely I must use a property that E acts like identity on the range of E W1 and it must act like the 0 operator on the perpendicular do you agree range of E is W range of E is W null space of E is W perpendicular okay and over range of E that is if X belongs to range of E then E X equal to X if X belongs to range of E then E X equal to X so E acts like identity on its range E acts like identity on W it acts as 0 operator on W perpendicular 2 conditions but you will get 4 equations in 4 unknowns alpha, beta, gamma, delta so we will determine E completely so let us do that in both the cases in the second case also this time we cannot talk about the orthogonal projection we will just talk about the usual projection it is called an oblique projection oblique projection so I will determine E in the second case such that E acts like identity on W1 that is Z1 and E acts like 0 on Z2 then I will compare these 2 case look at the structure of E how does E look like. So first E X equals X if and only if X belongs to W that is the range of E and so I am using this condition W1 right and E X equal to 0 if and only if X belongs to W perpendicular W1 perpendicular that is W2 in this example range of E is one dimensional W1 is one dimensional and span by the vector 1, 1 say I have not taken an orthonormal basis that does not matter span by W1 is span by 1, 1 so E of that vector must be itself so for one thing alpha, beta, gamma, delta operating on 1, 1 must be 1, 1 1, 1 belongs to W1 in fact that is the only vector in this only independent vector in W1. Yes it is V, V on to yes projection of V on to W1 yes I have this in the back of the mind that this will correspond to W1, W2 yes it is a projection of V on to W1 okay so I have this so this equation gives rise to 2 equations okay this matrix equation tells me alpha plus beta equals 1, gamma plus delta equals 1 okay 2 equations in 4 unknowns 2 more equations will come from the second set of equations E X equal to 0 so all the 4 unknowns can be determined so look at this E X equal to 0 if and only if X belongs to W2 I will take that vector 1, minus 1 equate that to 0 alpha, beta, gamma, delta into 1, minus 1 that must be this time the 0 vector 0, 0 okay that gives rise to these 2 equations alpha plus beta is 0 gamma sorry alpha minus beta is 0 gamma minus delta is 0 so alpha equal to beta equals half gamma equals delta equals half okay so let me write down E in this case are my calculations correct okay let us go to the second case now in the case when R2 is Z1 direct sum Z2 remember that there is I do not know if I have told you this before there is this terminology associated with perpendicular subspaces this E, this E is called the projection of V on to W1 along W2, E is the projection of V on to W1 along W2 that is what you have calculated along W2, E is the projection of V on to W1 along W2, along you can look at the geometry okay what is the geometry let us go back to the best approximation problem say I have a subspace pass through the origin this is X okay this is my subspace W and I want the projection of X on to W along W perpendicular so I must go along perpendicular that is all see this is perpendicular so I reach W along the perpendicular to W from X okay that is the reason it is called along W2 in this case it is the orthogonal projection but if you look at the case Z1 Z2 the second decomposition it is not orthogonal projection but we can still say that E is the linear transformation or the matrix of the linear transformation of R2 on to Z1 along Z2 we can still say along Z2 okay this time it is not an orthogonal projection, this time it is not an orthogonal projection okay so what I have tried to do is I have done the whole thing for orthogonal projections and I am trying to imitate this for the ordinary oblique projection I have taken the property of the orthogonal projection and trying to imitate what it satisfies in the non orthogonal case what do I expect of a projection in the non orthogonal case what do I essentially what do I expect the property of a projection that of course it is E square equal to E and it is a linear transformation okay apart from that what are the important properties range of E is W1 null space of E is W2 this is essentially what I am trying to imitate in the non or the oblique case and I am trying to see what is the difference between the matrices that I get what is the difference between the transformations in these two cases okay. So let us go to this case when I write R2 in this manner again I will do a similar thing so I will use these calculations I want E to act like identity on W1 which is Z1 and EX equal to 0 this time Z2 okay observe Z2 is not perpendicular to Z1 okay identity this equation will remain as it is okay Z1 is the same as W1 but the other one will be 1 2 okay this is the only change the other one will be 1 2. So I get these equations alpha plus 2 beta equals 0 gamma plus 2 delta equals 0 in the case when R2 equals this E is given by the matrix just avoid the confusion I do not want to write 2 E's so E is given by this matrix so what is that this time can someone make the quick calculation alpha is minus 2 beta beta is minus 1 alpha is 2 is that correct beta is minus 1 alpha is 2 the next one I will write without looking at the equations what is the next one it is not the same is it the same or minus of that it is the same okay the other see this will be a rank 1 matrix so the second row will be a multiple of the first row in this case the same as the first row this will be a rank 1 matrix you know why this will be a rank 1 matrix this is also a rank 1 matrix I mean this is not invertible this will also not be invertible okay in this case why is it not invertible no no please come on that is not my question you mean I will give you a 2 by 2 matrix ask you to tell me the determinant I want a qualitative answer E is not if you restrict the function from E to E then it is a function is bijective so then E is a linear transformation from R to R what prevents E from being a bijective linear transformation this is the question pardon why is it not an injective why should it not be that is the question can be expect E to be invertible that is the question can we expect E to be invertible yes that is the only situation that is the only situation only these are the only 2 extremes you expect so the only injective see E square equal to E if E is invertible what happens E is identity the other extreme E square equal to 0 the other extreme is that E is 0 now in one case V is equal to W so W perpendicular is trivial in the other case V equal to singleton 0 so W perpendicular is the entire space V these are the 2 extremes okay invertibility cannot happen for a projection if it happens it is identity identity there is no problem the problem is not there in the first place okay so E can never be invertible okay range of V equal to W so if E is invertible then W is the whole of V so W perpendicular is trivial okay so this is my E in the second case what is the difference between the structures of E is there any difference you can of course verify the D square equal to E in this example as well as in this example okay but what is there is there a qualitative difference between the structures of E in these 2 examples if yes what is it is there something that you can say for the first one which you cannot say for the second one structure looking at the form E transpose is E that is essential E transpose equal to E is in fact another characterization for an orthogonal projection E transpose not equal to E corresponds to the oblique projection so E square equal to E E transpose equal to E and range of E determine E completely E square equal to E E transpose equal to E range of E that is the subspace range of E must be known to me these 3 determine E uniquely in the orthogonal direct sum case in the oblique direct sum case I must know range of E I must know null space of E as well as the fact that E square equal to E in the oblique projection case E square is not equal to E sorry E transpose is not equal to E okay so remember there is another important thing which I hope you have observed which is for the subspace W1 there is a unique subspace which together with W1 forms a direct sum decomposition this unique subspace is unique in the sense that it is perpendicular to W1 okay but for the second setup Z2 is just one complementary subspace of Z1 in place of Z2 I could have taken span 1, 3 I could have taken span 1, 4 all these are complementary subspaces to Z1 okay so in the case of orthogonal projection there is a unique decomposition of the vector space V into W plus W perpendicular in the case of ordinary projection there are infinitely many direct sum decomposition that is given a subspace W there are infinitely many complementary subspaces that those can be obtained by an oblique projection those can be obtained by oblique projection the unique subspace is obtained by the orthogonal projection okay so this is really the essential difference between an orthogonal projection and an oblique projection okay so let us then move on to the next topic. The topic is linear functionals and adjoins, adjoins of linear transformations, linear transformations and adjoints of linear transformations, linear functionals is what I want to discuss first linear functionals and adjoints adjoints of linear transformations the notion of adjoint will generalize a notion of symmetry the operator the operation of taking given A taking the operation A transpose given A doing the operation A star that will be generalized in a inner product space we will call that as the adjoint operation but before that linear functionals okay so what is a linear function? Definition a linear functional definition underline field okay that is a linear function okay. What we will do is first give a representation for linear functions okay this result is easy and this also should immediately remind you of a theorem in functional analysis which you will do a little later. Linear functional and adjoint so first I have the following let V be a finite dimensional inner product space V be a finite dimensional inner product space and F be a linear functional on V so the underlying field is real or complex so F is a linear transformation from V to R or C then the representation theorem says there exists a unique vector Y in V there exists a unique vector Y in V such that the operation of this linear functional is like taking inner products with the vector Y such that the action of F on any vector X is given by the inner product X, Y for all X and V for a fixed Y if it is a linear functional then it arises precisely in this manner okay the proof is just by producing the vector Y okay and we will show that Y is unique okay but this is called a representation theorem that is any linear functional is represented by this inner product the proof is we start with an orthonormal basis and give the formula for Y let us say this is U 1, U 2, etc U n let V be an orthonormal basis of V, V is assumed to be finite dimensional so it has an orthonormal finite orthonormal basis I know F I want to find Y which I must show is unique and satisfies this equation. Define Y by Y equals summation J equals 1 to N F of U J bar U J define Y to be this vector F is known so I can determine Y look at the images of the basis vectors under F take the conjugates and then form this particular linear combination okay we will show that this Y satisfies this equation and show that this Y is unique consider for 1 less than or equal to K less than or equal to N the inner product of U K with Y this is inner product U K Y is this summation J equals 1 to N F U J bar U J the inner product is conjugate linear with the second one so when this comes out it goes with the conjugate double conjugate summation F U J into the inner product U K with U J with J being the running index K has been fixed so when J takes a value K this is 1 all other terms are 0 this is orthonormal basis so the only term that remains is when J is equal to K you substitute here it is F U K so we have shown F U K is U K with Y this is orthonormal basis so for any X this will be true because any X there is a linear combination right is that clear that is if X belongs to V then X can be written as summation J equals 1 to N in fact the coefficients are X U J with U J and so if you look at F X then F X is summation J equals 1 to N F is linear so it is a X U J F U J but F U J is I want to show F X equals X comma Y okay X U J F U J this is should have been better to it does not matter can I just say this is inner product X with Y again very you could have started with X with Y start with inner product X with Y then it is X into that formula F U J goes out and then I must take the inner product of U J with X so I get this so you have this representation for any linear functional F so there is a unique Y uniqueness we have to show suppose there is Z in V such that F X equals X comma Z okay then you can compare these two it follows that inner product X with Z is inner product X with Y this means that okay this means that inner product X with Y minus Z equals 0 for all X and V this means since this is true for all X I can replace X by Y minus Z which means Y equals Z so uniqueness follows immediately okay so every linear functional on a finite dimensional inner product space arises from the inner product with a particular vector observe that this Y belongs to null space F perpendicular which is not coming from the proof but this can be shown Y belongs null space of F perpendicular that is can you see why this is true proof let X belong to null space of F I must show that inner product of X Y is 0 then F X is 0 that is 0 equals F X but F X I know is inner product of X with Y so this means Y is perpendicular to null space of F if X is a random vector from F then Y with X is 0 we have shown so X belongs to null space of F V itself is finite dimensional null space of F is also finite dimensional so it has an orthogonal complement so I can write V as null space of F direct some null space of F perpendicular whenever you have a finite dimensional subspace it holds but V itself is finite dimensional so this holds so what this means is that a linear functional on a finite dimensional inner product space is completely determined by its action on null space of F perpendicular F is completely determined by its action on the subspace null space of F perpendicular of course the first term is null space of F so F takes the value 0 there F is completely determined by its action on null space of F perpendicular this is something which does not hold for a general linear transformation for a linear functional this is that is true okay. Using this representation theorem we will show how the conjugate transpose it is called the adjoint operator we will show how the adjoint operator for a linear transformation on a finite dimensional vector space exists and it is unique and also derive some of its properties okay that we will do next time.