 This video is part of a tutorial series just using Sympy, symbolic Python and Python in linear algebra, so it's kind of a quickies video, such as a two-minute Tuesday other than the fact that this is not Tuesday and this is not going to take just two minutes. But we're going to use Sympy and just look at the null space of real valued matrices. I'm using Visual Studio Code here, and it's a Jupyter notebook inside of Visual Studio Code, and you can see I've created an environment, a Python environment called Mathematics, and I'm running Python 3.9.12 for this environment. Just a little bit of background, so you can set up your system similarly. Now I'm going to import just this one package, Sympy, and as you can see I don't use namespace abbreviations, most of the times I don't. So I'm just importing Sympy, so I'm not using ASYM for instance, meaning that if I want to reference the functions and methods in that package, I have to use the full word. I'm also going to initiate pretty printing there, so I'm using the init underscore printing function so that we can get nice mathematical symbols printed to the screen. So just a little bit of background, because you need to have some understanding of linear algebra, some background, and have a little bit of understanding of vector spaces, and the span and basis vectors, etc. So I'm going to assume that you know that, but you just want to understand a little bit more about calculating null spaces, or if you have exam questions, they're always going to ask, you know, return or calculate the basis or basis vectors for the null space of a matrix. So that's what we're going to talk about. And the null space is indeed a vector space. It's a subspace, as we'll see right at the end, we're going to do a proof. And then it's one of those four fundamental subspaces. We have the column space, the row space, the null space, and then also the left null space or the null space of the transpose of a matrix. And what the null space really is, it's this space, subspace, spanned by these vectors x. So if we have this homogeneous system, matrix A times the vector x equals the zero vector, it's all those vectors x. Remember, that's the solution of a linear system is these vectors x. But it's all the vectors in there, although at least the space that they span. So those are the vectors that we're talking about. And you'll remember that we set up this kind of system to look at linear independence of the column vectors of a matrix. And that's exactly the same technique that we're going to use if you want to think about it that way. So we're going to have to understand linear independence there. That's, remember, that is where the one or at least a no column vector is a linear combination of scalar multiples of the others. So if we have this situation, A is a m by n, a rectangular matrix, x must be n by one. Otherwise, we can't define this multiplication, this matrix vector multiplication. And on the right hand side, then we'll have this matrix B, that is, will be a subspace, all those vectors B will be a subspace of the column space. So don't get those two confused and I'll space in the column space. The column space is all about this B. What constant multiple linear combinations of this matrix A of those column vectors, what space do they span? So what possible values could be take given, you know, any input value, the x, any solution x. So that's the column space, the null space is all about this vector x that we have here. The rank of a matrix, you'll see me use the rank of the matrix quite a few times during this video tutorial. And the rank of the matrix is just the number of linearly independent columns, column vectors in our matrix. That's the way that I want you to think about it. You might have heard about it as how many privates there are, but that's exactly what we're saying, how many linearly independent columns are there in our matrix A. And that's going to give us some idea of how many vectors we are going to have in the basis of our null space. And you'll see how that is going to work. So in this notebook, this is what we're going to be about. I'm going to show you all the possible matrices that you could get in a problem set that you might have to solve and that you have to think of. We're going to do square matrices and rectangular matrices. And for each of these, we're going to look at both the situations where the column vectors are linearly independent. And when they are, when there's some linear dependence. And with the rectangular matrices, you know, that really two kinds you'll have when there are more columns than rows or more rows than columns. So you will get both of those. And then both of those will look at the situation where we have linear independence and linear dependence. Now, if we have more columns than rows, there's a special situation going on there. So it's not quite clear yet what is going on there. And we'll just have to qualify when we get to that section. So let's look at the null space of a square matrix. Then there's a square matrix. And you see there the subtitle, the subsection title, there's linear systems with unique solution x for every unique b. So there's various ways that we can talk about this. And they mean exactly the same thing. If this matrix is invertible, then we will have that these columns are linearly independent of each other. If the determinant is not zero. So there's all, you know, all these statements that we can make that are actually equivalent statements as far as this matrix concerns. So I've created this matrix, I can tell you now the column vectors are linearly independent. And this is how we created using some pie. There is a matrix function that's uppercase m. So because I'm not using namespace abbreviations, it will be some pie dot matrix. And we pass a Python list to this function. There's my list object. And it has these three sub lists. So we enter these values row by row. So you enter them as row vectors one to one is my first row of my matrix A there two one three and then three one four. And that'll create this matrix for us. So we have to enter it in this way row by row. We're going to sign that to the computer variable A. And if we print A to the screen, we're going to see a mathematical representation there because we're using init underscore printing. So we see a nice representation of our matrix A on the screen. It is now a matrix some pie some pie matrix object, I should say. Now, when at the beginning, I said, you know, when we want to look at linear independence, we set up a homogeneous system and the augmented matrix of that, and we're going to do exactly the same here. So we're thinking about this homogeneous system. And remember, I said that the null spaces span by all these vectors x such that this homogeneous system holds. So what we've got to do to our matrix of coefficients here, which is all we have for a linear system at the moment is to add a column of all zeros on the right hand side. So we're going to create this augmented matrix for a homogeneous system. And one way to do go about that is to use this call underscore insert method. So my A is my some pie matrix object dot call underscore insert. And then two arguments, the first argument is what is the column index going to be of this addition, this added column that you're going to do. Now remember, Python is zero indexed. So this first column vector 123, that's index zero, 211 is index one, 134 is index two. So I want a third index. So it'll be three comma. And then I'm passing another some pie matrix. Now column vectors, just a matrix just as one column. So I'm passing again a Python list, and then each one goes inside of a sublist, each zero is going to go inside of a sublist. And I'm going to assign that to the computer variable a underscore augmented. And there's my augmented matrix. Now you know what we do from here, we do gas Jordan elimination, because we want to reduce the row echelon form. Very simple, of course, in some pie and Python a underscore augmented, that's my augmented matrix, some pie matrix object dot, and I call the RREF method, row reduced echelon form. And that's going to give me back two things. First of all, the reduced row echelon form of my matrix, and then the indices of the pivots. And you can see we have, you know, in our matrix original matrix of coefficients, we only had three columns, and we see the pivots are indeed in the zero of the first and the second column. So linearly independent, we can see on the left hand side here, we have the identity matrix. So that's another equivalent statement to say that the determinant is not zero, or it's in, it has an inverse or the determinants not zero, or you can reduce it to the identity matrix. Because we can read off from here that x sub one is zero, x sub two is zero and x sub three is zero. So we can both state the fact that there's linear independence of these columns, but we can also now think of the null space, because I do have a vector here, x sub one is zero, x sub two is zero, x sub three is zero. So I have the zero vector, the zero vector is in the null space of this matrix. The problem is the zero vector does not span any space, there's no subspace that it spans. So our null space is actually nothing, there's no null space here. Now remember I talked about the rank, that's the number of linearly independent columns, and we can see it right from here, there's three linearly independent columns. So the rank should return three for me. And I call the rank method, so a dot rank, so that's a on the original matrix of coefficients, dot rank, open close parentheses, we get back three. So we can write the rank of a equals three. And then remember for this matrix, it has m rows. So each of my column vectors, think about exist in three space. Let's go back to A, each of my column vectors, that's a column vector geometrically that lives in three space, I've got three independent vectors there, so they are a basis for three whole of three space. So this column space spans all of three space. And you can see there the m is three in our instance, three rows, and the matrix A spans the whole of that are three. So we're just talking about real valued functions here. And if we now talk about the null space, there's a very nice method, a dot null space, and you can see there's nothing there. The zero vector does not span any space, so it doesn't return this zero vector. But remember the zero vector is always there, it's always in the null space of every matrix. And then remember what I'm showing you here in equation two is that you read it off from your reduced row echelon form of your augmented matrix, x of one is zero, x of two is zero, x of three is zero, and we can write that in nice vector notation. Now we're going to move on to, that was the easy case of course, but I think it's important to understand that there is always the zero vector. But zero vector doesn't span any space, so my null space doesn't exist here. So this second example, I have the square space of a matrix with a linear dependent column vector. So I've generated this such that this third column is just the addition of one times the first column vector plus one times the second column vector. So that's quite easy. Remember that that does not make this third vector the linearly dependent one. I could also just say the first one is linearly dependent on the other two because I can take one times the third vector plus negative one times the second column vector, and that's going to give me the first column vector. So there is linear dependence whichever way you look at this. So the rank better give me back two. And it is going to give me back two. And if you think about it, each of these vector column vectors in geometrically, they are in three space. I've only got two of them that are the rank is two. So two of them at least are linearly independent. The third one, then any third one I can write as a linear combination of the others. If I only have two of these vectors in three space, I cannot possibly span three space. I'm going to span this plane through the origin. That's all I can do. I cannot span. So this column space of this is not the whole of three space in our example here. So let's set this up. Let's create the augmented matrix as we have before at my column of zeros with a call underscore insert method there. And I'm going to just do gas Jordan elimination and get the reduced row echelon form. And now we can see clearly here we've got a bit of a problem. We have all the zeros in the last row, as you would expect that to be. And we can see we only have these two pivots, zero and one. We can clearly see that from here that x sub three is going to be a free variable. Now you might have to write something like x three equals s, you know, put in these other variables, but we can just write it out as such. I can say x sub one from this table first row, this row reduced echelon form x sub one plus one times x sub three equals zero. And if I take x sub three over to the other side, this is what we have in equation three x sub one equals negative times x sub three. We can read off on the second row that x sub two is going to be negative x sub three. And x sub three here is a free variable. I've just have these bunch of zeros there. And I'm just going to say x sub three equals one times x sub three. So I'm not making substitutions with s or t. So if I look at it, I take x sub three out as a sort of common scalar there. So it's x sub three times negative one, negative one and one. And that vector now is in my null space. So I have a vector that spans the null space, any linear combination of that vector, any constant multiple of that vector. And remember, that's going to be a line through the origin of three space, any vector on that line. If I take that vector x, and I have matrix eight times any of those, I'm going to get the zero vector. So if you think about linear transformations in that way, you can see what the effect is, everything goes to zero. So I can use the null space method again, there. And I get back the same negative one, negative one one. And as I say, that is the vector that is now the basis of the null space, only this one. And there's only one vector in there. And that is what we refer to as the nullity. So you can see the nullity of a equals one, there's only one vector in the basis of this null space. And remember, we set the column space, the rank was two, and two plus one is three. And how many column vectors that we have originally, we had n equals or n for November and n equals three. And you'll always find, and that's a theorem that you can prove, that the rank plus the nullity will always equal the number of columns in your matrix. So in this instance, we see two plus one equals three. So that holds. In other words, once you know the rank, how many linearly independent columns there are, you already know, and you know, obviously how many columns there are in your matrix, you immediately going to know what the nullity is, how many vectors are going to be in the null space. And in our first example, remember, we had a consistent system there. In other words, we had a rank of three for those three column vectors. Of course, there was nothing left for the nullity. And indeed, we saw that was the case. So let's look at a null matrix, the null space of a matrix with more columns than rows. Now, as I said, we just have to qualify a little bit. That we that I said, we'll, we'll talk about linear independence. And then linear dependence. But of course, we cannot really talk about linear independence when we have more columns than rows. Think about it this way, I'm going to set up this matrix object, some pie matrix object, there we go. I have more columns than rows. If you just think about this carefully, each one of these column vectors on three space. And how many vectors do I need to to span or how many vectors do I need as the basis for three space? I have three, I only need three, but I have four. So even if the first three are linearly independent, that fourth one is not going to be, because I only need three vectors in my basis. So immediately, if I have more than that, again, that's a theorem that you can prove. But if you think about it logically, it should be too easy to understand. I only need three vectors on my basis. I told you all four, those are linear, or at least three of them are linearly independent, because I only need three in the basis. So the fourth one's always going to be redundant in the sense of a basis vector. So if I look at the rank of this, it's going to be three. It can't be more than that, because I only have three rows. So each one of these vectors only live in three space. In the rank, there is three. So there is linear dependence here. So even though I had the little subtitle of independence, what I referred to there, the qualification that I'm referring to there, each of these column vectors live in three space, exist in three space geometrically. And I have three basis vectors from these four. So I'm spanning that whole space. But most definitely, one of them is linear dependent on the other. I can write one as a constant multiple of the others or linear combination, I should say, of the others. So let's do our augmented matrix. And we see, there we go, and we can do Gauss-Jordan elimination to get it in row reduced echelon form. And we can see that we can do by hand, if you wanted to write it out by hand, you see there in equation four, we could do that. I want you to think about what the nullity is going to be. I've got four columns, and my rank is three. So think about that. So I can write from our row reduced row echelon form except one equals negative except three except four is also free variable. So I'm just going to put always put the positive except for the except two equals negative except three plus zero except four except three that is my free variable. So that's just equal to one times except three and except four is always going to be zero. There's no other value that except four is you can take. So I can read it off here. There's always a zero for the except four is zero coefficient. So we're not going to care about that. It's only except three that we care about. So it's negative one, negative one, one zero. Now look at this very carefully. Each of my column vectors in A existed in three space. But look at this, the null space here is in four space. I have four elements in this. Because remember, I have a rectangular matrix m by n m rows by n columns. My x has to be n by one. So there we go. We have this one vector in four space, and it's going to span a specific space, which we call the null space. And if we use this null space method on our matrix A, we see we get the same result negative one, negative one, one zero. So that is a basis for the null space. What is the nullity? It's one. And I'm pretty sure you understood that it had to be one. I have four columns, I have a rank of three, there's one left, the nullity is going to be one. And that's exactly what we found. By the way, if we just look at the column space, what it's going to do, it's going to work its way through these, you can think about working its way through all the column vectors, and it's going to return the first three that are linearly independent. And you can see at return three, that will be the basis of the space spanned by the column space of this matrix. And as we discussed, one of them has got to be a linear combination of the others, because in the basis we only need three, because there's three elements in each of these column vectors. And again, as I said there, the rank and the nullity is always going to equal the number of columns. So let's look at a matrix and then that does not span the whole of RM. So this is in the subtitle that I referred to with the linear dependence, but you could see we're always going to have linear dependence when it comes to a rectangular matrix with more columns than rows. What I'm actually referring to that is the spanning that whole space. So I've set up a new matrix here. And now I'm going to have the fact that there's only two linearly independent vectors in there. So if we look at the rank, we see that the rank is two. What is the nullity going to be? It's also going to be two. I better find two basis vectors for the null space. So let's create our augmented matrix just to look at that. There we go. We do gas Jordan elimination to get it and reduce row echelon form. And you can see I wrote it out there. Now X sub four is definitely in play. So I can write X sub one equals negative X sub three, negative two times X sub four. I can write X sub two equals minus X sub three, minus two times X sub four. X sub three is going to be a free variable. So I'm writing it as one times X plus X sub three plus zero times X sub four and X sub four. I'm writing as zero X sub four and one times X sub four. Because I have both X sub three and X sub four here as free variables, which means if I take X sub three out as a coefficient or as a scalar and X sub four is a scalar, I'm left with these two. And that's exactly what we wanted for the nullity. I have two vectors that are now a basis for the null space. And you can see they each exist, dramatically at least in four space. So if we look at the null space there, I'm going to get back those two vectors. And again, I see that the rank of A, which was two in the nullity, which is two equals four, the number of columns we had originally. Looking at matrices with more rows than columns. Now if we think of data science, this is a very common situation where the observations will make up the rows and the columns will make up the statistical variables that we have in our data set. And it's more common, obviously, to have more observations, more rows than columns. So let's start off by looking at a situation where we have linear independence between those columns. So I'm setting up matrix A again. And there's our SMPI matrix object. And you can see we have four rows here and three columns. So if we look at each of these column vectors, they would be elements of R four. They would exist geometrically in four space. And we have three of them. And you can imagine now if we just think of the column space for a moment, if I have four space R four, I would need four basis vectors to span that space. I just have three here. There's no way for me to span that whole space. Now I have set up these to be linearly independent. So the rank is going to be three. And there we go. These three column vectors are linearly independent. As I said, though, if you think about it carefully, there's no way for them to span R four, because they're only three, they're three basis vectors, but they are basis vectors for a subspace of R four. So let's look at the homogeneous system. We're going to tag on our vector column vector of all zeros on the right hand side, so that we have our augmented matrix using the skull underscore insert method. And let's do gas Jordan elimination. So we get the reduced row echelon form. And as you could imagine, the fact that we do have linear independence, if we look at the top three rows, at least we're going to get there, we're going to get the identity matrix, we see the indices of the columns for the pivots. And those are zero, one and two. So all three of our column vectors, they are linearly independent. If we just read from this reduced row echelon form of the matrix of the augmented matrix, we see that X sub one equals zero, X sub two is zero, X sub three is zero. Now remember, this null space that we're considering here is R three, because I have a four by three matrix, X has got to be three times one. So we only have X sub one, X sub two, X sub three, it's the zero vector. As I mentioned, every matrix will have a zero vector in the null space, but the zero vector does not span any space. So if we look at the null space, there, we'll see that it is an empty list that is returned. Once again, look at this though, I have three columns, my rank is three, of course, the nullity had to be zero, there was never going to be a vector in the null space or a basis vector in the null space then. Let's have a look where we do have linear dependence. That's a very dangerous situation to have in data science. So let's have a look at one of those and I use dangerous inverted commas. We have a simple matrix object there. I still have that each of my column vectors are in four space. And I only have three. So I can only possibly span a subspace of our four, even worse though, I do have linear dependence here. And you can clearly see what I did is I just, it's one times vector one plus one times vector two gives me the third column vector. So the rank is going to be two. So I only have these two basis vectors for the column space. So I can only span those two, use those two as basis vectors, and that's the space that I would span. So if we look at the column space method there, it's going to return the first two column vectors that it finds that are linearly independent. And then this instance is going to be the first two. As I said, they'd be basis vectors for the column space. Now I have that my rank is equal to two. I have three column vectors though. So I know the nullity has to be one. Let's investigate that. So we're going to set up our augmented matrix and we're going to do row reduction there until we have reduced row echelon form. And if we look at the null space, there is a single vector there. And by the way, in some textbooks and in some lectures, you'll, this will be referred to how many dimensions does it have. Now I don't like to use that term. I'd rather just specify the rank and the nullity and think about the fact that there is one vector that is the basis here for the null space and the two vectors that are the basis for the column space. I don't like to use the word dimensions there because I think it can introduce a bit of confusion. But I can read off of this reduced row echelon form of my augmented matrix. I see that X sub one equals negative X sub three. I see X sub two equals negative X sub three. And X sub three, of course, is a free variable. So X sub three equals one times X sub three. I take out X sub three as a scalar so that I have this vector negative one, negative one one. And remember this geometrically would be in three space. Although my column vectors were in four space. So just check up on that. And those are all the examples that you're going to find. Well, I hope all the examples that you're going to find. And now you really need, you really understand how to deal with all of them, what they mean, things specifically of the rank and the nullity. And if we think about linear independence, that's the same way that we're going to use to work out our vectors that are the basis for our null space. So lastly, let's just have a look at the proof. And we ask is this null space really a vector space? Well, if you think about a rectangular matrix a as m times n, the null space is a subspace of our in that's what we're claiming. So we know our in is a vector space. If you studied vector spaces before, you know that our in would be an example of a vector space that fulfills all those properties of a vector space. And we want to show that the null space is a subspace of this vector space. Now, to do that, once again, if you've studied vector spaces to show that a subspace exists of that vector space, you just need to throw show three things. The first of all, is that the null space or this subspace then contains at least one vector. And that we do know because we said it'll always have the zero vector. We want to know that the addition holds and the scalar multiplication holds. And by that, we mean if we add two vectors that are in the null space, is that addition of those two, the new vector that I find is that still in the null space. Or if I take a vector in the null space, and I multiply by a coefficient by a scalar, would that new vector still be in the null space. So we want closure under addition and closure under scalar multiplication. As I said, we already know the first one, we know that there's always going to be the zero vector for the homogeneous system. And now we have to consider two vectors in the null space. So let's make them x sub one and x sub two. And if they are vectors in the null space, they would fulfill this criteria x sub one equals the zero vector and x sub two equals the zero vector. That's given because it is stated that both x sub one and x sub two are vectors in the null space. And what we must show now, if we add those two that this will still be in the null space. Well, if you look at the left hand side, I can rewrite that as x sub one plus x sub two, which we have here at the end of the line. And each of them, we were given the fact that each of those equals the zero vector and the zero vector plus the zero vector plus the zero vector is the zero vector. So we have closure under addition. Let's look at closure under scalar multiplication. So given again, that x is a vector in the null space, we must show that a constant multiple of that scalar vector multiplication would still be in the null space of a. Well, that's also very simple because a cx, as you can see there, the scalar times the vector, I can rewrite that as cax and ax equals zero, the zero vector. We are given that because x is in the null space. And c times the zero vector or any constant scalar times the zero vector still the zero vector. So we have that scalar multiplication also holds. So that is the null space. I think you have a very clear understanding now of what the null space is when it refers to these matrices. Very easy to use some pie just to check up on your work. And when you get to a stage where you don't have to do this with pen and paper anymore, it's so easy just to use Python.