 In this notebook we're going to talk about hypothesis testing. So we're really building on what we had in the previous notebook where we saw that we can create this population and we can sample from it. And that sample is going to give us a distribution. We saw the law of averages and as much as the larger sample size, the more it's going to just represent what the population looked like. But we also had this critical thought about a sampling distribution. If we could do our study many times over, there's going to be a distribution of our actual findings. So we're going to build on that. We're going to talk about the scientific method. And the scientific method is all about hypothesis testing. So notebook 08 hypothesis testing. Let's look at the packages that we're going to use. Of course NumPy pandas and the stats module from SciPy and the plotting library is nothing new there. So the scientific method, really the basis of how we conduct our research and the basis of data science. And we're going to do that through this process called hypothesis testing. And that's the knowledge we're going to build. We're going to develop that knowledge based on what we saw previously, first about randomness and then about sampling. So the first example, we're going to run through a couple of examples. And the first example we're going to look at is a sample based on proportions. So a few scenarios that we're going to have in this notebook and thereby develop our intuition about hypothesis testing and the scientific method. So consider here a population and they have two mutually exclusive traits. Trade A and trade B. So if that population has trade A, that individual, that sample, that subject does not have trade B. And when it has trade B, it doesn't have trade A. So they're mutually exclusive. And we know for a fact that in this population, 27% have trade A and the remainder, the 73%, they have trade B. And our population size, our total population size is 3000. So whether it's some astronomical object, whether it is some organism in a laboratory, it doesn't matter what it is, we have a whole population of only 3000. And we know for each individual subject in that population, that subject will either have trade A or trade B, trade A occurs in 27% of the cases, trade B occurs in 73% of the cases. We know that. So let's simulate that. And what we're going to do is we're going to create this population from this. And I'm always going to call it this theoretical distribution that we set up. Okay, so let's create a population. We're going to seed the random number generator with an integer 42 by choice. And then we're going to call numpy.random.choice. And it's going to be trade A or trade B. I want 1000 of those. But now we see a new argument, P equals, and that's probability equals, because we've got to add some weights to how many times A is chosen at random versus how many times B is chosen at random. And we have to have something that sums to one, and it's going to represent this theoretical distribution of ours, 27% and 73%. So let's imagine that and we generate our population. And from that, we create a data frame object. So we're just passing that as a dictionary. So key value pair, the key is going to be my column value. The value of my key value pair, those are going to be all the column values. So we see trade B, B, B, B, A. So of course, the Bs are going to be more than the As. Let's look at, remember the unique method for a pandas series object, df.trade. So that column unique, we get Bs and As. If we do the value counts, we can see for our population, 2187 had B, 814 had trade A. And if we set normalize equal to true, we see it's very close to the 73%, 27%. So great, that's what we want. We can visualize this data of our population, a bar chart, remember, the nominal categorical variables, gap in between our bars, indicating that we're dealing with a categorical numerical variable. So imagine now we have a sample and we find relative frequencies of 0.13 and 0.87. And this is to a bar chart of what it really is in the population versus our sample. So we can see there's a difference, but can we put a value to that? Can we say, can we express how different this is? And whether we think our sample is still a representative of the whole. So let's do the following. Well done, what we've done before. We're going to take a random sample of a hundred instances from our population. So that's all we're doing. We're taking a hundred from there. And we see there's our hundred A, A, B, A, A, B, B, A, B, B, B, B, B. That's our hundred. Okay, let's do the numpy.unique function. So not calling the unique method, we're calling unique function here. We take a random 100 of these and we also return the counts. So it's a little bit different from using the unique method here. We're using the numpy.unique function because what we're going to get back is the unique values A and B and also how many there were. So in this instance, there were 30 from, with the trade A and 70 of those hundred had trade B. So that's just a random, we took a hundred. So we can count this. So look what gets returned an A and a B and a 30 and a 70. That's what unique with the return underscore counts equals true argument. And that's what it gives us. So what we're interesting in is discounting this 30. If we do this over and over again, and we count how many, how many times we find trade A. So if you think about it, Python is zero indexed. So this would be the first, the zero with, I should say, and this is the first. And so from the first, I want the zero with element just that 30. So if I can do this many, many times over and I just take how many times there are the straight A, I'll get a distribution of the possible trade A's. And we can see now our 13 that we got from our actual experiment, you know, what, you know, with that 13 have occurred very commonly or would have not. So what I'm interested in is then that first value. So just let's run that and we use indexing. So we're going to use first index one. That means it's the second lot that the unique function returns. And then in this second lot, I want the first one. So that gives me this index zero. So I see this time around, I got a 26. And we can imagine again, we can do this over and over again. And we can see where our 13% that we got for our result, where does it fit into the bigger picture. So let's do that with a for loop. I'm going to create this empty Python list. And I'm going to set my sample size to be 100. So now I'm going to run for I in range 5000. So from 0 to 4999. So I'm going to do count dot append. Remember what the append method does to a list. It just keeps on adding the value and the value that we want to keep on adding is we do this selection of 100. And we return the counts and we interested in just that a trait. So indexing one zero. And we divide by how many they are. So we've got to do that division by the sample size. That gives me the proportion. Every time this code runs, every time for this 5000 times that for loop runs, I'm going to add another element to my counts list. And it's going to be the proportion of a's. That's all I'm collecting there. So let's collect those proportion of a's and then this plot this out as a histogram using go dot figure graph objects dot figure. And there we see a distribution of trade a proportions. And they on the left hand side have done a little line of our 13% trade a. And now we can see our 13% would occur very infrequently. It has a very small likelihood of having been being possible. Given that we know the population and we continuously random sample from the population. So once you understand this, this is one way to approach this problem in a very powerful way to approach this problem. Original question was we know the population parameter. We know the proportion. The proportion is a parameter of the whole population. And we take a sample from there and now we get a new proportion of trade a of only 13%. That's a statistic. And we want to know is our statistic, how does it relate to the population? Is it a likely proportion to have found what we can do and to express how unlikely or how likely our 13% was as we can do the following. We can repeatedly sample from the population. And what we capture every time when we repeatedly do that is one of our statistics. And the statistic we're interested in is the proportion of trade a because we found 13% where we want to know how likely was it to find that 13%. And now we see, we see that it is very unlikely. It's way down here. So what we can actually do is the following. We can say sum up how many times remember our count is now going to hold a long list of 5000 proportions. How many times was that less than 13? So we're using a conditional there less than 0.13. But that count is a list. And if we want to do this, we've got to change it into a NumPy array. So I'm passing it as argument to NumPy dot array. So now it becomes a NumPy array. And it's going to be true, false, true, false, true, false, et cetera. And remember that we can sum over trues and falses because trues is, that's a 1 and falses is 0. So we can sum over all of those. So we'll find out exactly how many times it was 13 or less. Another little subtle hint I want to give you here. We're talking from negative infinity here to this 13%. Well, not negative infinity because the minimum would be 0%. So let's say from 0% to 13%. I can't ask anymore, what was the exact likelihood of 0.13? Why can't I do that? It's because my proportions here are a continuous numerical variable because it can be 0.13, 0.13, 000, 0001, 0.13, 000, 000, 000, 000, 495. So I can't express this as a single value anymore. Any value for a continuous numerical variable is just a round off. So we can no longer look at just the probability of a single value. Remember when we rolled the two die and we summed them up, we asked for probability of 10 or more. So here we're asking what was the probability of getting a 13% or less. So I've got to have this interval that I want to express. So now I'm asking between 0 and 13%, what was the likelihood? And that's this little gap here. So if I do this, that should give me the proportion of times that it would be 13 or less. That's the only thing I can truly now express. So that is one way to tackle the problem by this repeated sampling. So I just want you to understand now that this idea of repeated sampling exists and we're going to build on that, build on that until it becomes a very powerful thing. By the way, instead of using randomness here and uncertainty in our values by repeatedly resampling, there's of course a statistical test we can do with the chi-square test for proportions and it exists in the stats module of sci-pi. So I can just say stats.chi-square. I can pass my observed values as a list, my expected values. So I got 0.13. I'm just multiplying it by 100. So it's 13 and 87. That's what I got. And theoretically I expected a 27 and a 73. I'm just doing that just to show you just the numbers of that. And I get this power divergence which shows me a statistic of 9.9. That's my chi-square statistic and the probability of having found that. And it's not quite what we found, our open 004, but what we can say is very, very small. I mean the total probability of maximum is 1. Minimum is 0. And still a very small probability by this test. And what we saw here as well, between 0 and 13, very unlikely to have found such a result. So this idea of repeated sampling can give us an idea of how likely our results are that we find for our study, for our research. Very important idea. So let's keep on building and we're going to do another example. And this time our statistic is going to be the difference in means. So let's set up a little scenario as well. So in this example, we know the value of a continuous numerical variable in each subject in a population. And the sample space elements are on this interval, semi-open interval 0 to 100. So 0 is included, but it goes to 99.99999. Okay. So this distribution of elements, we say it takes on a uniform distribution in the population. So every value between 0 and 100 has an equal likelihood of occurring. So let's set up a population like that. So using the numpy.random.random function. We haven't seen that one before. So what random is going to do, random.random, that function, it's going to take a value on this interval 0 to 1. And it's completely uniform. So every value in between 0 and 1 is equal likelihood of being chosen. And I'm just multiplying it by 100. So broadcasting there. So every value between 0 and 1 that it chooses for me. And by the way, I want 3000 of them. That's the population size multiplied by 100. So now we're going to get this, these 3000 values between 0 and 100. Now let's further imagine that our population is spread between two neighboring towns. And a researcher suspects that there's a difference in the value of this variable between the two towns. Now, not having access to all the known values as we do, we know the exact values. We've just created that population. Of course, our researcher doesn't know this. And this researcher goes and they sample 100 individuals from each town. And the result they get is mean for this variable of 45.3 in town A and 52.8 in town B. Now, we know that there was no difference between the two because we created the population. But this researcher goes out, takes 100 samples from each and gets those results. Now, how can the researcher assess the difference between these two? We know what the truth is. The researcher does not. All the researcher knows is that's the mean in town A and that's the mean in town B. How can this researcher express this difference in means? Well, the difference in means means we subtract one from the other. That's going to be our statistics. So you can take one town 52.8 minus 43.7 and see there's a difference in mean for this variable of 7.5 between the two towns. But there's no reason why we favor one town over the other. So we might reverse this subtraction. So we could also get minus 7.5. We can subtract the towns in the other order. So we have this idea of the difference being 7.5 or minus 7.5, depending on which way we do the subtraction. So let's do the following. Going to repeatedly sample. So now we're getting into this in a thought experiment where we could repeat this experiment many times over. So we're going to recreate code. We're going to repeat the sample and then we're going to visualize distribution of our test statistic being the difference in mean. So let's set that up. I'm going to have an empty list difference and then I'm going to do this a thousand times over. I'm going to sample A. That's a random choice from my population of a size of 100 and a random sample, another one, choosing another 100 from my population. Now if you can imagine, I should put a B at the end there, we can now imagine that if we just draw randomly from a sample, we have this idea that we don't care when we randomly sample in which town they are. And we're going to get to why that's important. We don't care what town they are in. So we're just taking 100 random subjects and another 100 random subjects. And we're going to calculate a mean for the first sample and a mean for the second sample. And we're just going to subtract one mean from the other. And we're going to append that to this empty list of ours. So in the end, we're going to have a thousand differences in there. And we've just chosen sample A minus sample B. But remember, we could also do sample B minus sample A. But we've got to choose one of the two. So there we go. We have our 1000 differences. Let's have a look at our 1000 differences. We're going to do a histogram of that. So there's our histogram of all, you know, 1000 possible differences repeatedly sampling from the whole population twice and subtracting one from the other. And what we see in red and green there is A minus B and B minus A. So that's our researchers with with their negative 7.5 and 7.5. And again, we can now ask how likely was it to find those results given if we did this, if we could theoretically do this over and over and over again, you know, what would it possibly be? And again, I want you to start thinking it'll be the probability of this negative 7.5 and less plus the probability of this 7.5 and more. So how many of all of these were minus 7.5 and less and 7.5 and more? That is how we're going to express the probability of this what this researcher found. But we've set it up in a very unusual way in that we know what the population really looked like. And now we see this researcher comes along and that's real life, isn't it? I mean, the values do they are in whatever subjects we have for our study, they the whole population would have a certain value. But as researchers, we only take a sample of those. And we have to somehow express how likely it is that we have found the results that we find. Yeah, though, we've simulated from repeatedly taking from the whole population. Now, we understand that very well now. We've seen two building on the previous notebooks, we've seen two examples, here one of proportions and one of difference in means. There is a way we can attack this problem in data science. We can repeatedly sample, give this distribution of all the possible values that it could take and we somehow fit the one that we have in there. We have to take another step though, and that's coming up a little bit of this in this notebook and more in the following notebooks, that we have to now take that only from the sample that we have, because we don't have access to the whole population. And I'm going to show you how to do that. First day, let's have this idea of hypothesis testing. And that is really the bedrock of the scientific method in data science and on all sciences. And that's this idea of a null hypothesis and an alternative hypothesis. And we've actually done just that in these two examples, but we now have to explain what happened. So in hypothesis testing, when we design our research, we have to have hypotheses about what we're going to find. That is our based on our research question that we have. And we're going to have two hypotheses. One is the null hypothesis. And that's the one we accept because we don't have any data yet to show any kind of result. So we always have to take the null hypothesis, the null road, we don't know. And then we're going to collect data and we're going to measure that. And we're going to see if this value was indeed befitting our null hypothesis. And that's not really true, but let's use that to those terms now. Or it was such an unlikely event that we can go to the alternative hypothesis. So there's always going to be these two, a null hypothesis and an alternative hypothesis. So let's look at that first example of ours. We said that the proportions of trade A and trade B would be 0.27, 0.73. And our null hypothesis is going to be exactly that, that that is the proportion 0.27 and 0.73. And our alternative hypothesis is that it's not going to be 0.27 and 0.73. It's as simple as that. We have a null hypothesis and we have an alternative hypothesis. So we'll actually break it down a little bit further. We'll choose one of the traits and we'll say a null hypothesis is that the proportion of trade A is 27%. And alternative hypothesis is no, it is not. All that remains is for us to draw a line somewhere. When is, how much must it be away from 27 before we no longer accept this null hypothesis and we accept the alternative hypothesis that it isn't 27. Is it 26 and 28? Or is it 20 and 34? Where is that cut off? And I've already shown you how. Somehow from the data that we have at hand, we can draw this distribution of our sample statistic and the value that we found is going to be either very likely or one of the unlikely ones. And that's exactly how it's going to work. As I said before, the only step we have to take is that we don't know the whole population. We have to somehow use only the samples that we have to draw that those curves. So this idea of a null hypothesis and an alternative hypothesis. I just want to be very clear about the alternative hypothesis as well. There are actually two types of alternative hypothesis. And the one that we use most often is a two-tailed alternative hypothesis. And what we said there with the proportion being a null hypothesis that the proportion is 27. The alternative is that it's not 27. So it can be less than 27 or more than 27. It goes both ways. And we had the subtraction one from the other. And we can do the subtraction in either order for the difference of means. So we have to consider both sides of that draw. And that's a two-tailed hypothesis. Which means we also get a one-tailed hypothesis and a very dangerous thing to do. But it does exist and we have to talk about it. Now we have to have this idea. Again, let's discuss that how different must it be before we move away from our null hypothesis and accept our alternative hypothesis. And that's a very arbitrary decision. And you've all seen it when in scientific subjects where we still use the p-value and those are getting less and less. We have this cut-off. It's called the alpha value. And very often it's chosen at 0.05. So we'll say that if our proportion, if the proportion of the values that we found up to that extreme remember, we went from 0 to 0.13 in our first example. And we counted how many times that happened. And that was less than 0.0004. So that's very small. We put this cut-off of 0.05, for instance. That's an alpha value. And we say, well, if it's less than that alpha value, we will reject our null hypothesis and we'll accept our alternative hypothesis. But it's this line in the sand that we draw to distinguish between when do we decide that our results are far enough away from the null hypothesis that we will now reject. But please see that it's completely an utterly arbitrary. As human beings, we decided on that 0.05. Sometimes it's 0.01. And in physics, in high-energy physics at the Large Hadron Collider, it's much, much, much smaller because they have so much more data. Their values are going to start approximating the theoretical distributions, or at least the real population out there because they've got so many samples. In the life sciences, maybe we have 30 samples, 100 samples, 200 samples, not nearly anywhere near the whole population. So we can't go for those small values. We go 0.01 or 0.05. That proportion of values up to that 1.3, or when we did 0.75, minus 7.5, I should say, and 7.5. Let's go back to that graph. So we can count the proportion of these differences that were less than minus 7.5 and more than 7.5. And if we combine those, we can say if that was more than 0.05 of the sum total, we cannot reject our null hypothesis. If it is less than that alpha value, and say, for instance, it's 0.05, we reject the null hypothesis. But that 0.05 is just a thumbs up. Somewhere on this graph, we drew two little lines, and I'm going to show you those two little lines. And we said, well, if our findings are beyond that, we reject the null hypothesis. If it's inside of those, we fail to reject the null hypothesis. Listen, very carefully, we can never accept the null hypothesis. We just fail to reject it. And I'm going to show you how we build on that because we only have our sample values. We don't have the whole population's values. How we then say we fail to reject the null hypothesis, because the distribution that we're going to draw by repeatedly sampling only from our values that we have not from the population, we're going to get this graph based only on our samples, not on the whole population. So we can never accept it because it's based on the null hypothesis. So we would say we fail to reject the null hypothesis. But if we find something extreme out here, we reject the null hypothesis and accept the alternative hypothesis. But listen again, we're going to draw this graph only from our samples. And our samples are going to be based on the null hypothesis, a distribution of or under the null hypothesis. And if we find something in there, we can only fail to reject it because it's based on it. So you can start to see what is really behind this kind of data analysis, how we express if there really is a difference or there is not a difference. It's very arbitrary, very, very arbitrary. We chose that 0.01 or 0.05. And that's called our alpha value. So let me just show you a little bit of the technicalities, the notation that we use. So for null hypothesis, we would usually write h, uppercase h sub zero. And then we would say our random variable one and our random variable two. And we put a little bar over that, that means the mean. So you're going to say for our variable x in group one, it's mean equals that very same variable in group two, it's mean. So it's the same variable. We're measuring the same thing in both groups, taking the mean in both groups of that very same continuous numerical variable, and we're saying the mean is equal to each other. And our alternative hypothesis can be written as this little alpha at the bottom, h alpha. And our two-tailed alternative hypothesis says that the two means are not equal to each other. Now remember, if we subtracted those means that would mean the difference is not zero. Here we would say h zero is the difference is zero, h alpha, the difference is not zero. That would be exactly the same thing as they, and we choose this alpha value as 0.05. So let's do, let's run through another little example. So we're going to imagine that we have two populations here. And in one, we did some intervention, gave them a new drug, or we had some new psychological intervention that we performed on them, and the other group, we did nothing. And that's called our placebo group. So I'm going to set up these two, because we don't have access to real data, we're going to simulate some, and I'm going to use the stats.norm.rvs. Norm means that it's going to have a normal distribution. With a mean of 50, a standard deviation of five and 100 individuals, please, and I'm setting the random state so that when we run this code, we get the same results. And our second, our placebo group for that very same numerical variable, we're going to have a mean of 48, a standard deviation of seven, and also 100. So I've got my two groups, an intervention group and a placebo group, and they are 100 individuals in each group. So that's what we have. Now in statistics, which is a part of data science, there's an easy test to do, it's called student's tea test. I'm just going to show you how very easy that is. So I've got my two sets of individuals now. And I'm going to have 100 values and 100 values. I just want to say, are they different from each other? I do student's tea test past my two lists and I see a P value of 0.0168. And that would mean if we would to do this repeated sampling somehow, we don't know yet. We do this, we're going to get a nice lovely graph. We're going to see what the difference is in the means between these two, subtracting one group from the other and in reverse subtracting them. So we have a positive difference and a negative difference. And that's going to fall somewhere on the graphs. And we're going to calculate how many times it was more extreme on either side of those. And we'll see the proportion of those that come out are going to be very close to this P value, 0.016. And if our alpha value is 0.05, this is a very rare finding. And what we're going to say is we're going to reject our null hypothesis and we're going to accept our alternative hypothesis. There is a difference in this variable between these two groups. So these two groups, they are really different. So I'm just doing a little print statement here just to show you I have the mean for the one group. So we just took that from those theoretical distributions. So we have an actual hundred subjects and 100 subjects. So for the intervention group, we had a mean for this variable of 47. And then the placebo group of 49 and then placebo group 47. And with our t-test we just did, we say that that difference between 94.5 and 47.2, that is a statistically significant difference. That would be the terms that we would use. And we would say that the proportion, if we could do this over and over and over again, the difference that we found would be a very rare difference. Under the null hypothesis that there really isn't a difference. Our null hypothesis is that the mean of the one minus the mean of the other should be 0. We artificially created these with slightly different means. That's how we set it up just to simulate a research project. And we came up with 47.5 and 49.5 and 47.2. And the t-test here shows that that is a significant difference. Let's look at the box and whisker plot of these two. You can start building some intuition because we're not going to just do statistical test here. We're building a much deeper understanding of what's going on here. So there's our two groups, our placebo group and our intervention group. Somehow in the intervention group the mean of this variable that we mentioned was higher than the mean and the placebo group. And by the test we saw that there was a difference. So I just want to show you just to remind you that we can subtract one from the other. We get 2.21, but we can reverse that order and we get negative 2.2. So depending on how we do this, one can be more than the other. So we're really talking about a two-tailed alternative hypothesis here. And what I want to show you just in this little graph here, I worry too much about the code in here, is what the t-distribution looks like. I said, you know, we did that test and was a student's t-test. And this is what the t-distribution looks like. And this is, and let me explain through what this is. I mean, it looks like a normal distribution, doesn't it? It's nicely bell-shaped. But it's a distribution that we use in cases where we don't know what the whole population is. We only have data from our research project. The 100 samples and 100 samples, that's all we know. We do not know what it is in the population. And that's very common. That's how most research is done. In that case we use the t-distribution. And what the t-distribution shows us is all the possible differences that there could be. And it is only based on the sample size and how many groups we have. So we have two groups and we have 200 people. And so we just subtract 200. We subtract two from 200. So the total sample size, we had 100 subjects and 100 subjects. 200 minus 2 is 198. And this t-distribution only depends on that 198. It's the parameter for that distribution. And we call it 198 degrees of freedom. And I've just used that with stats.t.pdf and mp.lin space. Don't worry about any of these. I want to show you what this distribution looks like. And in the next notebook, we're going to simulate all of this. I just want to build up your knowledge. So this is what the t-distribution looks like. And we found a difference. Remember our researcher went along and found a difference of 2.21 or negative 2.21. So that's the difference they found and we somehow have to plot that on this graph of all possible outcomes. But we can't just put our differences there because our difference is dependent on the units of the variable that we measured. And the t-distribution doesn't care about that. So somehow we've got to convert the units that we measured into this t-distribution. And we do that by that little formula there. It's the difference in mean divided by the square root of the standard deviation or the variance at least in group one divided by the sample size of group one plus the variance in group two divided by the sample size. Don't worry about this. That's how you convert your difference in means to an actual t-statistic so that we can plot those differences here. And that's what I've done for us here. Don't worry about the code. We don't care about any of this at the moment. We just want to see what it looks like. We've converted our difference into a t-statistic, a t-value that we can actually plot on this distribution. And I want to remind you again, this is a theoretical distribution of all the possible differences that is possible. And this is the difference that we found. But we have to remember that we could also have done that in reverse. It could also be the placebo minus intervention, or the intervention minus the placebo. So we've got to convert that difference on the other side as well. And how are we going to find out if this difference that we had was likely to have been found? So our null hypothesis is that there's no difference between these. And you can see the t-distribution. It's very likely to have found no zero differences versus this differences that we actually found. Way out here, or our research, are found. And what we're going to do is to just calculate how many times was it less than this? Add to how many times was it more than that? If we figure out those proportions out of the whole, that is going to give us a p-value, approximate a p-value. How likely was it to have found this result? And we have some magical decision of 0.05. And if it's beyond that 0.05, we reject the null hypothesis and we accept the alternative hypothesis. So don't worry once again about the code here. I just want to show you what that would look like. So what we have here in these two middle lines, they are called critical t-values. So those would be the values if we find beyond that, we can reject our null hypothesis. If it's inside of that, it was one of the more likely differences to have been discovered, given we can do this over and over again. But ours was outside of that. And what the mathematics of it does behind the scenes here, it actually, the total area under this curve is one. And it's just working out what the area under the curve is from this purple line towards the negative side, from the orange line towards the positive side. And it'll be 2.5% of the area under the curve on one side, 2.5% on the other side. So the total is 5%, 0.05. That's our cut-off value. And what we're working out is the area under the curve from our finding towards this side, from our finding towards that side, add all of those up and that's going to be less than 0.05. It's beyond these critical values that would denote 0.05. 2.5 on either side, symmetrical, we divide it by 2. And that is for us, how we discover whether we fail to reject the null hypothesis and accept the alternative hypothesis, or we fail to reject the null hypothesis and that's it, we stuck with it. Or we found one of these rare findings, then we reject the null hypothesis and we accept the alternative hypothesis. And in some fields of science we would say we have a statistically significant finding here. So I hope your intuition, your understanding is building, building, building. We're not there yet, we've got the next notebook to go. So this last section is really optional. I'm going to talk to you about one-tailed alternative hypothesis and something we almost never use. You've got to really convince your peers that you knew beforehand that one group was going to be different from another. So you can skip this part, it's purely there just for completeness sake and we're going to talk a little bit about one-tailed alternative hypothesis. So let's remind ourselves of the two, as we can see there, the two means for our two groups and the intervention group was 49 and 47 in the placebo group. So let's do the following. Let's make group one, our placebo group and group to our intervention group. So this would be one alternative hypothesis that we state that in our alternative hypothesis, that group one, the placebo will have a mean less than the group, the intervention group. And that's exactly what we have here, 47 is less than 49. So a null hypothesis really is that they equal or the placebo group is greater than, this first group, the placebo group is greater than the intervention group. So that's our null hypothesis, either equal or greater than. The alternative hypothesis is that it's less than. And if you see that, that's actually what we found. But now in the context of this one-tailed alternative hypothesis, do we find a result? So I'm just going to show you, we'll just calculate the critical T values there. And then I'm going to generate this plot. And there we go. And we see our critical value there. Now it has shifted a bit to the right as when we had both. We're not reflecting it on both sides. We're suggesting that this is the critical T value. If it was less than, you know, it will fall somewhere here, but still our difference fell here. So it's still a tiny little fraction of the whole, if we think about how many values fell here versus the whole. So we can really reject this null hypothesis. Remember, the null hypothesis said that these two means would be equal or the placebo group would be more than. The alternative hypothesis is that it was less than. And indeed, we found one that it was less than and actually that was such a big difference, it fell way out here. And what would we do if it was the other way around though? Now I know hypothesis says that it's equal to or less than. So that's really what we expect. We expect it to be less than 40. And we had that the 47 less than the 49. And our alternative hypothesis is that it's greater than. So we're going to do critical values for that. And I just want to show you the graph. So don't worry. Once again, in almost all of this notebook, not to worry too much about the code. That's about the concepts here. So what we have now is our critical value way out here. And this was the difference that we found. But what we are looking for now in the clues in this little line of code here, it's one minus. So it's the whole minus this little bit. And that gives us this whole bit. So our probability is actually quite large. It's actually quite large. And if we now go back to these, to this, as we're not going to reject this null hypothesis, we fail to reject the null hypothesis. In actual fact, this placebo group had a mean lesser than that Sydney didn't fall anywhere near the greater than. So if that's the way that you have your null, your null and alternative hypothesis, remember, then it's one minus this little bit. Or if it fell here somewhere, it's one minus this, the whole minus this. So we're actually looking at this positive end. But the way that it works, it counts from this negative end. So we really, we really just interested in this whole area. So that's the two ways that we that we consider the the one tailed alternative hypothesis.