 In this video I want to help you to understand what a p-value is when we do research. Now I'm going to use the Wolfram language inside of Mathematica to show you what it's all about. So don't be worried about the fact that I'm using a computer language. It is not about the computer language. I'm just using it instead of using PowerPoint. So no PowerPoint slides. I'm going to show you some code. It's not about the code. I'm not trying to teach you how to write the computer code. It is only used instead of PowerPoint for instance so that I can show you what a p-value is all about. Now I've chosen Mathematica in the Wolfram language though because it is a fantastic language to do your statistical analysis in and then also to demonstrate the meaning behind things. How things work. So I'm not going to show you. It's not about showing you equations of the p-value. I want you to understand what a p-value is so that when you do your own research, when you read a paper you can understand what a p-value is and specifically what its uses are and what its shortcomings are. I'm going to start off by doing something completely different though. I want us to go back to school and remember what the equations are for the areas of a shape because believe it or not that is what a p-value is. So let's just start with that. Now I'm here inside Mathematica and I'm going to show you just a few of the reasons why I always choose Mathematica above say Python or Julia. There is so much built into Mathematica. There's knowledge built into the language. It's a fantastic computer language. One of the reasons why I absolutely love it is instead of writing code I can write a normal English sentence. I can click here and say I want a Wolfram Alpha query and I'm simply going to type in area of a square. So I did nothing special. I didn't write any code, nothing like that. I hit enter and it's going to query the Wolfram Alpha database and give me all this beautiful knowledge about the area of a square and that's just me asking a plain simple question. So there we have a square. Remember the two sides are equal to each other and if I have a side A and the other side A, the area of the square is just a squared. You remember that from school. That's not a problem. Now it gives you all sorts of other information about the areas of squares. Look at that. It's all fantastic beautiful. I want you just to think about the area of a square though. Let's forget that computer language for a minute although I do love it. So let's look at the area of a triangle. Can you remember that? There you go. You can see how to calculate the area of a triangle and you see if the sides are A, B and C exactly how to do that and again all sorts of extra information. Let's go on to something that's a bit more difficult. You remember the area of a circle and there we go. If the radius is A, the area of a circle is pi A squared or pi radius squared, pi R squared. Let's do one more. The area of an ellipse. Now believe it or not we're working our way to understanding the p-value. So here we have a major axis, a smaller axis here and we see that if that is x and y or A and B at least the area is then pi times A times B. Simple enough. All sorts of extra information. Let's try one. Let's try one more. Alpha, Wolfram Elf Aquarium. I'm going to say what is the area of a trapezoid. Let's see if we can do that and there you go. If those are the areas of the trapezoid, there is your equation for the area of a trapezoid. All sorts of extra information once again. Let's bring it back to a p-value though. Believe it or not the most important shape that we are going to deal with here is the area of a rectangle and that's quite simple. Let's see what it is and there you go. If we have sides A and B the area of a rectangle is just A times B. Very very simple. Let's move down through all the information we can get from the language and now knowing that we are talking about the area of a shape let me build your intuition for what a p-value is and we're going to start off by just taking a simple die. You know the thing that has six sides in your rollet and one of the sides land up face up so it has sides with values one to six. I think most people know a die. These are discrete values though. Discrete meaning I can only roll a one, a two, a three, a four. I can't roll a five and a half with a single die. That doesn't exist so these are discrete values and I'm going to take that die and I'm going to roll it 1,000 times. I'm going to simulate that with a very easy piece of code. As I said remember this is not about the code but this is this little line that you see here. That's how easy it is. I'm doing this experiment rolling a die 10,000 times and I'm saving it as a computer variable. Don't worry about that. The next line of code is what it's all about. Let's execute that and that's called a bar chart. So you see at the bottom here one, two, three, four, five, six so that is what I rolled and if I just hover over there you can see how many times you know this is completely random. Well it's actually called CEDO random but that's a different story. I rolled it at random 10,000 times and it seems I got 1,586 times it landed on one, 1,660 times it landed on two, 1,708 times it landed on three. What you can notice though is that it's you know this was random so it's almost exactly the same. One more thing I want you to notice is if I added all these values well it had to add up to 10,000 because I rolled it 10,000 times but let's just express this bar chart as something else, something that looks slightly familiar, slightly similar but actually means something different. It's called a histogram. Now you see this histogram same sort of thing I see one, two, three, four, five, six at the bottom but instead of the how many times it was you know how many times it occurred when I rolled my die it says what was the probability of landing on a three, on a four, on a five, on a six. Now let's go back up to one you see it was 1,586 and lo and behold well let's choose this one there we go. 1,660 times it to occurred and if I go down I see the probability was 0.166 well that's almost similar or it seems all I did was I took how many times it occurred the 1,660 and I divided it by 10,000 because that is the probability of it having landed on two and remember the area of a rectangle that is just the base times the height we looked at that that was just a times b now these are discrete values remember I control a five and a half so the base there of the two what we are saying to ourselves when we are talking about just just go with me here a little bit that we are saying that the base is equal to a length of one it is discrete now even if it was something that was smaller than one it being discrete means we take it as a unit length just one and the height is 0.166 of my probability histogram here and 0.166 times one equals 0.166 and that is the probability the p value I can ask you if I roll to 10,000 times what was the probability of getting a two well it was 0.166 and that was the p value it's the probability of something having happened having done your experiment you notice there it is simply geometric area that is just a p value it is a it is just a geometric area as simple as that now let's ram things up a bit or at least let's just do something that is not random there's actually an equation for what is happening here and it's called a probability density equation or probability density function it will tell us very nicely what each of these p values should be now remember this was done at random and everything being equal all of these should have been exactly the same and there's an equation for that and it is this equation that we see here the probability density function it says the probability x of hitting anything between a minimum and a maximum so that's a and b the minimum that you could roll was a one the maximum that you can roll was this b which was a six so the probability of hitting anything between these so a one a two a three or five four five or six was one divided by one minus a plus b so let's just do that so it's one over one minus one plus six and that is just one over six and I can ask Mathematica to express that in for me in approximate form so that's 0.166 so the p value for each of these theoretically should be 0.166 because it's there's equal likelihood if my if I'm if there's nothing wrong with my die or the way I roll it or the floor it lands on the table it lands on that should be the probability what I want to you to see here though is I can create a mathematical equation called a probability density function that allows me not to have to draw this but to just work out what the probability would be and that is you know we're going to build on that and that eventually is the equation from which we get a p value after doing a normal medical or other kind of study notice that I can develop a function that can represent the solution for me now let's do something else I've still got my die and now I get another one out now I've got a pair of dice now we can call it dice I like saying with dice as opposed to a die that just sounds so so weird anyway so I've got my pair of die or my dice and I'm going to roll them both I've got two of them and I'm going to roll them 10 000 times I can't do that but I can ask a mathematician to do that for me and I've saved in this variable called dice and I can say let's look at the first what was the first 10 rolls and you see this is called a list of lists I rolled a three and a two next I picked it up rolled a five and a one picked it up rolled a four and a three picked it up rolled a four and a one picked it up rolled a six and a one picked it up rolled a one and a six rolled it up picked it up rolled a six and a five that's my first 10 rolls now let's do something let's just add each of these so it'll be three and two is five five and one is six four and three is seven four and one is five et cetera so I'm just going to total up each of those rolls now how many so I mean I could can roll a 1 and a 1 that's 2 and I can roll a 6 and a 6 that's 12 so it goes from a minimum of 2 to a maximum of 12. Now which one which one from 2 to 12 do you think occurs most commonly? I think naturally you would reason to yourself well it's I mean you've rolled a pair of dice before you know that two ones or two sixes at least is very rare to hit two sixes much more common to do the others. So is there some pattern here? Now remember all I did was I rolled two things that have equal likelihood of any of the sides you know lining face up so what is going to develop here? So let's have a quick look let's tally all of these and it says I rolled 277 twos, 588 threes a bit more, even more fours, even more fives, even more sixes, even more sevens but when it gets to eight I start going down again 9 less still, 10 less still, 11 less still and 12 you know even less. So let's do a little bar chart of that and see what happens. There we go there's our bar chart and look at that is that not one of the most beautiful well this is discrete it's not continuous values and again I can't roll a seven and a half these are discrete but isn't that the most beautiful bell-shaped curve you've ever seen. So I take two things and each of them has equal likelihood of occurring but if I somehow if I do something to them and this instance we add the two values up I get this distribution that is beautifully normally distributed it's bell-shaped and once again I can express this as a probability there we go it was 1,687 and the probability of rolling a seven was 0.1687 just how many times I rolled it was 10,000 and I take how many occurred 1,687 divided by 10,000 that gives me the p-value but notice also I can ask you what was the probability of rolling a 12 well it was less than 3% it was 2.85 percent a p value of 0.02 so notice how the sum of all of these add up to one so you can't have something occur like a 13 you know or a one so it's got to sum all these probabilities I've got a sum to a one just as the occurrence of all of these have to sum to 10,000 the probabilities have to sum to one and I could also say what was the probability of throwing something that was 10 or more while I would just add these three I would add 0.08 0.05 then 0.285 I could add all of them so I can color in these three at the end and I can say well I can draw a line down here and my probability of a 10 or more will just be the addition of these three so again dealing with discrete values very easy to calculate a p-value it is nothing other than geometric area of rectangles again discrete so even though it works out nicely that these are you know the difference between 10 and 11 is just one but because it's discrete we always we always take a value of one as being the base so the base times the height here the height here we can see there clearly the height of this little one for instance is 0.0285 times one equals 0.0285 and that's the probability it's the geometric area of a rectangle as simple as that now let's have a look at something that's more continuous not the step size discrete value with a base of one because we've seen a bell shaped curve that's definitely not stepwise it's a smooth curve so what do we do there I tell you now it's not a problem let's have a quick look just as I had my equation there for the probability density function of rolling a discrete uniform distribution that is 1 2 3 4 5 6 and each of them has a uniform or equal likelihood of being you know landing face up I can make a probability dense not me you know someone someone very clever you know a while ago came up with this equation and that's the equation for a normal distribution with a mean there of mu and a standard deviation of sigma and that's what the equation looks like one of the most beautiful equations you'll ever see and I want to give you some intuition of how this thing looks and works so there we go I can manipulate how much the mean is so I can move the mean around first of all look how beautiful that's a beautiful normal distribution and beautiful bell shaped curve and the area believe it or not underneath this line which goes from all the way to negative infinity all the way to positive infinity the area of this probability density function the area under this curve and to really do that we need integral calculus you know not just the area of a rectangle that was simple base times height yes something different we need integral calculus I'm not going to show you the integral calculus I want you to appreciate the fact that the area under this curve is also equal to 1 and I can also get a little piece here in the corner or wherever and I can calculate what the area under the curve is let's just increase the standard deviation a bit and you see how it gets flatter but it gets better on the sides that the tails move out and they get thicker on the side and you can see what the influences of changing when the influence of the mean is easy to see you know it just goes left and right and then the influence of the standard deviation on the bell shaped curve there if this was exactly one and this was exactly zero we would call that the standard normal distribution anyway so I just want you to appreciate the fact that we can go from discrete values to continuous values and most of the time that is what we are going to look at I just want to I'm going to move away a little bit from what a p-value is because I'm quite interested just to show you a couple of other distributions this is the t-distributions you know students t-test it is also bell shaped but it's slightly different and it works not on a mean and standard deviation but it works on something called degrees of freedom so here we have degrees of freedom of two and degrees of three freedom of three and you see the two there you see how the tails get a bit fatter there or it gets a bit thinner and it gets pushed up but that's the distribution you usually use in a t-distribution not to worry let me just show you the chi-squared distribution chi-squared distribution also works on something called degrees of freedom and here we see one with two six and twenty and you see a chi-squared distribution with a degrees of freedom of 20 almost has a bell shaped curve but when it gets really small like a six year or the two year it's certainly not bell shaped okay so now let's create a population and instead of there's been seven billion people we imagine we in a town or some area that services some institution whatever there's 20,000 people and imagine there's some blood test call that the value for that blood test is called the variable remember variables and let's just assume that this value that this blood test can take so if I were to take all those 20,000 people and I were to do this blood test on them there's an equal likelihood of all of the different values that are possible being available so it's again a uniform distribution in the population but this blood test has a minimum value so the patient that has the lowest value is a 50 and the patient that has the highest value has a value of 110 just chose that now really means if we selected two groups at random from this population so I have 20,000 the student in front of me and I take 30 and I take another 30 they shouldn't really be a difference between the two means so if I take this city and I calculate the mean and I take another city and I calculate the mean and I and I compare the means I mean there was this norm there was equal distribution about across the possible values so they shouldn't be a difference should they let's see so let's use my population I'm suggesting what I'm saying to the computers take values between 1510 and give me 20,000 of them but every value has an equal likelihood of being chosen let's have a look at the histogram the histogram of that and here you see the fact that it is uniform so all the values don't worry about this one over 110 years just what happens when when these graphs are generated what I want you to appreciate the fact is that if I if I make a little bundle now what has changed here and what makes this the fact that these are not no it's not one at the bottom anymore is that there could be a value of 50.1 50.2 50.435 67.876 and what we're doing is we're just burning them we're saying well let's go from 50 to 55 exactly and we see how many people were in 50 had a blood value between 50 and 55 and we divide that by the 20,000 which gives us a probability to fall in that range I cannot ask the question any longer what was the probability of someone having exactly 50 because they could be 50.01 50.0001 50.0001 could have been the blood test these are now continuous values and I have to bend them now I have to say between 50 and 55 exactly how many people were in they divided by the 20,000 still gives me the probability but look at this there's an equal probability of landing in any of these little intervals I want you to appreciate that now once again I could there's a little equation just to say what the what the mean is and remember if we look at 110 to 50 halfway between those two is 80 so theoretically it should be 80 and if I do the theoretical calculation here you can see that it is 80 but for the 20,000 that came at random it was about 80.2044 and you see again there you know the many decimal values so we're not we're not talking discrete values anymore we've moved away from that but there's an equation just to look at what the theoretical mean should have been and what our actual mean is now so so bear with me I mean this is this is crucial that these there's equal likelihood of any of these little intervals you know of a patient falling in any of these little intervals it's not that one occurs more commonly than the other now what I want to do is something similar to rolling those those the pair of dice I'm gonna take I'm gonna do a very simplified study I'm going to take two people at random from the 20,000 and I'm going to look at what each of their blood value is and I'm just going to instead of adding them I'm going to look at what the averages so one had a blood value of 50.8 and the other one had a blood value of 104.786 I add them together I divide them by two and I see you know what was what was the average and I and I take my two people and I calculate that average and I write it down and let's imagine I throw those two people back in the bunch of 20,000 now I select two again might be might be that you know I picked the same two one of the two is the same but there's 20,000 probability is I'm going to pick you know two other people and I just do this 25 times every time every one of those 25 times I calculate the average of the two that I took and now let's have a look at what happens to the distribution of these averages I mean intuitively you should think well it should just be flat like this curve is flat because everything is equally likely so if I were to take two of them and took their average all the averages should be equally likely hang on a minute let's do that whoa what has happened here we can see a pattern developing here it's almost like a little bell shaped curve wants to come out here now appreciate this it was equally likely for anything you know they all equal equal likelihood and I just at very random chose to and did the average but lo and behold it seems as if some averages are more common than others let me prove this to you this is called the called the central limit theorem now what I'm going to do is I'm going to take 30 patients and record the average and I'm going to do throw them back take another 30 I'm going to do that 500 times and each time I'm going to record I create an empty list day and every time I run through it I'm gonna run do 500 times I'm gonna go 500 times choose 30 calculate the average spit it out let's have a look at that there we go so definitely some averages were more likely to occur than others and this is what happens when you do research you're not gonna do it 500 times over you are only going to do it once you don't have the time and money to do a study 500 times over and that one time that you did it your study you got a value some values should occur more commonly than others some should occur rather rarely and that is our P value what was the probability of finding that one that you did and look at this this is based on the fact that there wasn't this skewed distribution of this value in our population to start off with it was very equal it was very equal so I you know I didn't do anything behind the scenes to to cheat the system and forced the fact that I'm going to get these people were equally distributed here and still I get this bell shaped curve out now again I can use that beautiful equation I showed you up above and I'm going to calculate the mean and not the standard deviation but what we you what we have to use is the standard error and let's just look at this distribution here which I've I know I've grouped people into little buns so little intervals there let's do it smoothly so I can just show you what the theoretical distribution looks like and there we go beautiful beautiful bell distribution and you could do your study that 30 that you picked to land it there and so say landed here if I hope you can see my cursor I can now ask the question what was the probability of the group that I got the average being this value or higher that will be the area I can draw a straight line down and the line towards this side and this little funny shaped area I mean it's like a triangle but there's a curve to the one side so I can't use the equation for the area of a triangle I've got to use integral calculus and it can calculate this area under the curve and I could say well the probability of having found a mean of my three patients of so much was of so much and more remember this is not discrete I cannot ask the question what was the probability of getting an average of 80.5 I can say 80.5 or higher or 80.5 and lower and that means I draw a line up at 85 here and I check what was the area under the curve to the left or the area under the curve to the right but it's the same principle as those rolling of the two dice it's just area under the curve now let's just go one final step here and relate it to something that you would actually do so you would take two groups of 30 people and you would compare the means one against the other so we take two people 30 values on one side 30 values on the other side I get a mean on this side I get a mean on that side I've subtract the means from each other and now I want to see is there a difference in the means and this is what I've done in this line of code so I've done that and learn behold I still get this nice distribution but now it's around zero zero is more around the zero area I can't ask exactly zero remember around zero the difference of zero but in the means between the groups should occur more commonly so if it comes out here towards this side I can ask five six seven eight nine so what is the probability of getting a difference in means between my two randomly selected groups of 30 people what what is it to get a difference of means of say nine or more well I would just add these but once again we will not do it in this blocky form we can use the actual the actual equations theoretical equations that we have for that and there's our beautifully a beautiful curve now it's not going to be exactly around zero because this comes directly from my running my experiment 500 times but I think you can appreciate what is going on here that given that there should be no difference given that some deity knows that there's no difference I can construct an equation which will give me this beautiful shape curve and I run my study once and I find a value and that means I can ask the question what was the chances of finding my value or more than my value my the difference in my means or more than or less than or different than doesn't matter all these questions but it in the end I draw a little line my value my one study falls some way down here and I can get an area under the curve and towards the one of the sides let me let me show you that in these two little graphs that I've drawn here in a different program and I just imported it here so imagine then you do your study and you've take 30 people and one in each group and you know one group got the certain drug and the other one not and you measured something and you want to see is there difference now in these and the means between these two values and what you would do now remember you can't do this a 10,000 times over you can only do this once but what those equations will do is they'll take the one that you got and it'll draw this theoretical nice little curve with a total area of one underneath you will then do the following you will say I want a p value of 0.05 and the equation will actually work out for you well if I draw a line right here this orange one and everything out towards that this area is behind the blacks imagine the black wasn't there so all the area of the orange there that is 0.05 of the area under the curve it will work that out for you now you'll take your specific results and and you'll there's a little equation that you do to mark out where you will then put your dot here on the x-axis and will fall there and it'll work out for you what this area under the curve for the black is which is obviously a less than 0.05 so the probability of you having found this specific result given all the possible ones that could occur theoretically your p value was less than 0.05 now this is called a one-tailed hypothesis most commonly in research though we'll have what is called a two-tailed hypothesis where we say we think our null hypothesis is there's no difference between the groups our alternate hypothesis is there is a difference so if I subtract the mean of the one group from the other the one might be more or less than the other one I'm not saying specifically beforehand that I know and that is really the way you should do most research when you have a one-tailed where you say I definitely know the second group should be more and you have a one-tailed hypothesis you must be able to to logically explain to your colleagues who's reading your research and they convince them through logical arguments that you should have chosen the one-tailed so it's much more common to do this two-tailed so the two-tailed says the null hypothesis is there should be no difference and I think I've shown you now the way that these things are constructed is the fact that there was no difference there was an equal distribution of a uniform distribution of these values it wasn't like there was one lot you know more people had 80 than 50 that was equally likely to have everything so the null the p value comes from the fact that in reality there is no difference between the two groups given that fact you now construct this nice little graph that I've shown you and some differences will be more common than others and you just find one of the uncommon ones then you have a small p value that that never never ever that this prove that the one group was different from the other you never prove that there is no equation that can prove that this is built from the fact that they should be no difference statistics like this is both from the fact that they should be no difference in reality then given there is no difference some deity knows there's no difference you just found one that you should result that would let less likely to be found given that there is no difference and then we decide okay we'll reject the null hypothesis and accept alternate hypothesis and say there is a difference between the groups but that's just something we decided we never prove the statistics cannot this inferential statistics cannot prove this it is based on the fact that underlying those patients that you took there really is no difference you didn't then find the difference that should be rare to find and we call that oh okay oh there is a difference between them well I think you understand now now you have a deep intuition about what a p value really is it is a geometric area based on the fact that on equations that were developed for there to be no difference whatsoever it is just the fact that we as humans have decided that if we get one of these differences that is less likely to have occurred then we'll call it significant and we had this cut off of 0.05 so just to get back to this so what we do is we don't have the one is going to be more than the other we actually draw these two orange lines with slightly further out and we have 0.025 or two and a half percent on the one side and two and a half percent on the other side and whatever of these two black values we find we just duplicated on the other side and we just add the two black areas to each other so it'll always be double what it is so your p value you know if there's one way to cheat just go from one tail to tail to a one tail because you just divide your p value in two and suddenly it'll be much smaller of course it'll be half and you can cheat that way which you should never ever do and that's a horrible horrible so when you read a paper see what kind of hypothesis people chose one-tailed or two-tailed alternate hypothesis and if they chose or if they don't say to that with some suspicion and if they do say and it was one tell they better convince you why the one tail should have been chosen so we do this two-tailed thing and because I love mathematical let me just show you so I'm just going to create 30 people two groups 30 people and from a normal distribution I'm going to choose a value with a mean of a hundred and standard deviation of 10 and the other one of five one fifteen let me just show you at random what happened here so the mean of my first group was 99.867 the mean of my second was 103 I can ask you know let's just look at a box and whisker plot you know what those look like and I can ask is there a statistically significant difference between these two groups couple of a couple of functions you can use in Mathematica the one is called the location test I do it there and I see there's my t-statistic that is the little black line that's where you draw your little black line there it takes your values and converts it into a point that you put on the x-axis you then reflect it on the other side with a positive as well and it works out the p value for you 0.31 so given only this information it uses those beautiful equations that we saw before draws that nice little bell shaped curve and it says well the difference between these two was minus well minus four something around there and that means on that x-axis that will fall on negative 1.01935 so that's so many standard errors away from zero don't worry about that duplicate it on the other side work out there another curve and learn behold it says look buddy this area under the curve was not so uncommon to ever could the difference in the means were not so uncommon given all the possible ones that could occur and let me just show you there's another way just write the t-test and say group A and group B very easy and Mathematica to get these p values very easy to do so I hope this has given you some intuition about the p-value of what it really means it's just this geometric area based on the fact that there isn't any difference and we are weird as humans we just decided that 0.05 is gonna be this cut off and then we think there is a difference you've never proved it you never ever ever proved it this is just the way we constructed statistics clever people constructed statistics and I think we abuse it so you can you can really see start to to get this uneasy feeling about a p-value and why some journals are moving away from it and how easy it is to cheat with a p-value and how little it really means and there's some as you know the other things that we should look at when we do but you're going to see a lot of p-values and you should know what it's all about