 Maybe we should go ahead and start Can I so you guys I was it I have to admit it's been I think I was asked to give this talk like many months ago So I've forgotten the context a little bit you guys are resident a residence in ophthalmology And we'll do a research rotation as part of the residency is that the the idea Yeah, okay, so I'm so that's probably why I'm here talk to you about doing statistical analyses as part of your research rotation and then later you know later in your career the I should mention that I'm the director of the study design and biostatistic Center and the population health Research Foundation, which were basically research infrastructures to support Research here at the Health Science Center And you guys have access to to to our center for various kinds of research support so I'm going to talk a little bit about that and then I'm going to talk about some topics in Study design basically And just the whole process of doing research where statistics comes in The so I mentioned the study design of biostatistic Center. It's in the population health research foundation This is a part of the University Center for Clinical and Translational Science And we started in 2008 this was a small little group of four statisticians There's a lot of research needs here at the institution. So this is now grown up to a big Multidisciplinary group that provides research support and it actually includes not just statisticians, but also psych epidemiologists Psychiatricians qualitative researchers Health economists So basically it's the whole group of people who are who are expert in research methodologies related to statistics But other quantitative health science type disciplines We kind of work together to provide it, you know overall I'd say quantitative health science research support and We the center is largely funded by having separate having arrangements with different departments and divisions But this includes the Department of Ophthalmology So you guys actually have access to a certain amount of free support from the center And I should mention stop me anytime if you have any questions, it's not the kind of thing you have to wait to the end So this population health research foundation Just to give you an idea it's structured with these five Cores which includes the study design and biostatistic center that I just mentioned cancer biostatistics health measurement survey methods health economics and systematic review The systematic review has to do with Working with the library to do systematic literature searches that might support meta-analyses for example But we provide support in all these areas And it's a pretty broad-based sort of thing. This includes clarification of research hypotheses This is actually something that is very important in research Just clarifying the question In fact that in many ways is the really the most important step And we can work with you in a way I mean we could have make sure that the questions being asked or the kinds of questions you can address statistically And the questions that can be you could actually have Address feasibly within your resource constraints But I'll talk come back to this a little bit more. This is actually a key step I mean then this leads to developing study designs Key element of the study design is how many patients what sample size or if it's in basic science how many animals This then evolves into a statistical analysis plan It's always important and a key step whenever you would work with us is before you actually do analyses It's writing a protocol for those analyses. It's very much like the idea of doing a research protocol before doing a study Before doing data analyses. It's important to actually write out what those analyses are going to do be and write out a plan for that And we'll come back to why that's important in a little bit And in the group we have people who could help you with data collections survey design If you're developing instruments for measurement, which is often I know in a lot of our work with Ophthalmology in the past. This has been a key step Actually developing questionnaires or appropriate methods for measuring your outcome variables And we can help with this so I mentioned we have a group that does systematic reviews This is actually often a good step before embarking on research Make it it's a pretty obvious but making sure you do a good literature review always And sometimes that should be elevated to the status of a true systematic review Which is just basically a protocolized literature review that assures that your literature review is unbiased It's not cherry-picked that you really are getting a comprehensive review of the literature Not missing us, you know certain perspectives Our group could actually carry out that analysis If you don't want to do those, but if you prefer We also have people in the group who specialize in supervising people who are clinicians who want to do their own analysis so we can work either way The and I mentioned the economic analyses with our economic core, which is getting more popular these days I've been focusing on quantitative methods, which is really what most of the group specializes in But there actually is a sub-discipline within this that's called qualitative research It's just usually the idea of doing focus groups qualitative sorts of review In situations where you may not have quantitative outcomes specified And then of course we could help you with writing the papers and so forth The areas of expertise are listed there. I won't go through these in great detail But I guess my the point of all this is just that we do have this resource available And I would just encourage you guys To to work with us if you have statistical needs or needs in some of the other areas that I've just mentioned I'd say a key a key thing is to come soon enough I know if you're going on a research rotation, that's a challenge because that's a very limited time to do research I've never really quite understood that phone that particular paradigm But you need to come to us very early on so we can get engaged with you at the beginning stages of the research So we actually have time to go back and forth. It's a very iterative process And it's not a process where you come at like the last stage and just ask somebody to do analyses Our work needs to be integrate integrated in with the research from the very beginning Okay, any questions about this part of this sort of thing? Okay, so next I'm just going to go through a What could be a typical collaboration process? Where you would you know, how how would we work together if we if we are doing work to get you know Helping you with a particular study And as I mentioned this will start with the actual planning the study developing the research questions conducting the literature review and Starting to think about the study design and potential outcome variables Basically thinking through the framework of the study And this is the stage where we would like the collaboration to begin. It's working through this Then the next step for you is Well actually these could go hand in hand, but at some point you need to fill out a request a Collaboration request form and you can just go to this website and you'll be a little form to fill out to spell out what your needs are And I guess actually I probably should have put this before this This stage we'd like this to come first It's not if you can do a little bit of literature review first go that's we like that so those first meetings are more productive but if Even this stage will be an interactive process going back and forth Then we'll have meetings we'll work together to refine study design data collection and analysis Variables and then this will eventually lead to the development of a statistical analysis plan Which is an actual written document that will lay out the the analyses that we'll be doing We then ask you as the principal investigator to approve the analysis plan by email And this is basically means that you've agreed and committed to a particular analysis plan And then finally we would they'll be carrying out analyses implementing the statistical analysis plan Finalizing results and then that goes and leads into the final manuscript The starting your study some of the key elements of that are here are listed here I Don't want to over if I can't overemphasize the importance of the thorough literature review to begin with I know when we've worked with junior investigators before this step Sometimes isn't done complete fully and then that really just leads to a lot of it can lead to wasted time And in particular use a design that wouldn't Development of a research design that might not be optimal for your situation or doing work That really doesn't need to be done with some at some variation is really what should be done At this stage you may be considering alternative designs some of the issues are listed there and It'll be important as you go through this to start thinking of outcome variables predictor variables For example a key outcome variable may be mortality. I guess it ophthalmology usually not mortality, but there'll be other outcomes related to Visual function that will be this will be be focusing on those that will need to be clarified You may need to be thinking through predictor variables or grouping variables If it's observational research That means without randomized, you know not a randomized study With observational research, it's a key step is thinking through factors that may act as confounders These are variables that are associated with both the predictor exposure variable and the outcome you're studying And you think these things through and then the workout strategies for controlling for these variables And then if relationships you're studying may vary in differing groups of patients Those are called effect modifiers We would need to think identify those ahead ahead of time as well The request form is here and I mentioned the website before it's actually a pretty short form And it just you provide background on your project What your basic objectives are? If you work with a statistician already or have somebody or use would like to work with You can suggest the statistician on this form and if they're available, we'll go ahead and assign that person Filling out the form gets you into our tracking system Which assures that we don't lose track of the project the absolute worst way of contacting our center is the email me I not that I do this intentionally, but I get so much contact that I can sometimes we lose track of it important very very very very important are the Timelines and making sure we initiate the collaboration process early enough in the game to actually get the work done Okay so in meeting with As we meet with a statistician or a other member of the population health research foundation will be Refining as we mentioned the research questions and the hypotheses refining and developing the study design That includes things like working out the inclusion and exclusion criteria the analysis variables confounders Clarifying what the limitations are which is important to acknowledge up front And then working through how data will be collected And I'll mention in this step in data collection. It's often under emphasized the institution and our center are strongly recommending not using Excel Which is the has been a favored method in the past for collection of data Excel is a spreadsheet that doesn't provide a basis for tracking data changes And it has just a whole variety of features that lead that kind of promote errors being made And Instead it's much we prefer that data be collected using a true database And the database which is being used for research now and supported throughout the institution is red cap And I would recommend you use that The bioinformatics core in the center for clinical translational research Supports the use of red cap and provides training and can help you get set up using it It's actually very very easy and it basically is structured in a way as to promote research and and promote research quality data Now When we're meeting and developing the analysis plan The best mode we've discovered for communicating analyses is really going right and visualizing what the Tables and figures are going to look like in the eventual manuscript. And so if it's possible whenever whatever possible It's we find it preferable to actually Develop mock-up tables that would at least summarize the structure of the tables and figures that you're envisioning in your paper This really speaks things up when we can develop this Okay, any questions about any of this? Okay Now I just want to mention why we're emphasizing the importance of this collaboration going from the beginning we've there's been a lot of in the last decade or so attention paid to a Concern that much research that's published in the medical literature is is actually non reproducible and or and that the findings that are reported may actually be false and This is one paper which has been kind of dramatizing this this this concern But the concerns relate back to Issues of inadequate sample sizes Where the study was done, but the power calculation really wasn't done right and the study was too small to answer the research question that was being addressed Another very common issue is Not really understanding the size of an effect that would be expected or plausible Or that the power should be power or that the study should be powered on And a miss this is really getting back to sample size again, but very often there's a kind of mismatch or you know Between the effect size and the sample size or the effect size that are targeted are just unrealistic So we will do a lot of work there Major issues have to do with absence of these pre-specified Hypotheses or absence of pre-specified analyses So the challenge here is is that when analyses are just done post-talk And I do know a lot of groups work this way We've seen it a lot you could be like the abstract deadline day comes along and you just start pouring through data looking for a Significant p-value to put in an abstract that kind of work is just horribly non reproducible You it's a kind of post-talk data dredging that really is not research and And I'll show you in a minute some examples of why though if you come up with a significant result in that way You really can't expect that if you were to do a new study later You would get that that result would reproduce And so this is a Why it's so important to before you collect your data before you do your analyses to spell out what your analysis plan is the This is related to the issue of the really the more relationships that are evaluated the more you test The greater the possibility is that you're going to find some false positive results in all these relationships And then another key thing is Being precise about the research you're going to do as opposed to kind of leaving things vague to be worked out during the Analysis step and so what our population health research foundation tries to do is we try to help you through all these steps to avoid these errors Okay The statistical analysis plan, which is really key and this is again why the Collaborations need to start early versus late. So we have time to develop these and think them through carefully Have been more and more recognized as absolutely fundamental It's really for maybe decades. It's been clear that Analysis plans are required for randomized studies and in fact without analysis plans many journals will not published Research from a randomized study They all often ask you asked to see the analysis plan just as proof but in the last Decade or so there's been more and more attention to Developing in analysis plans not just for randomized studies, but also for observational research Okay So I mentioned there's a stage where we ask we try to we get approval of our statistical analysis plan via email And this you know, there's approval process. There's a couple reasons for it. I mean one is just logistics This will help assure that we don't waste time on analyses that are not what you intend to as the as the principal Investigator, so it's making sure you close that loop, but it's also represents a commitment to a particular analytic approach That's being spelled out before doing the analyses Now we do it sometimes you do discover You come up with findings during the analyses that suggest additional Additional analyses this does happen and it's not we're not saying we won't do these additional analyses. We will but we will keep note of what analyses were post-hoc and We're done and basically because some other result in the data suggested that they should be done as opposed to analyses that were Prespecified and so that process will get documented and we'll be able to know what was Prespecified and what was post-hoc and that distinction is important and actually needs to be expressed in the publication of the of the research Then finally all this once analyses are done Then our attention will come to the actual publication process itself. So I mentioned we can work with you on this Okay, so that I'm just wanted to go through there basically what the process the research process is of working with our center How to contact us? Timeline issues Any questions about any of that? Yes Yeah It's free and it is I don't think you're you're actually working on its web based Yeah, so you're it's not necessarily that you're downloading it to your own machine, but you're reworking on the web Any other questions And it's very easy. I mean I think that just about it's always a One-hour little training session is always sufficient for a typical study coordinator to learn it and actually then Set it they can and you can then set up your data set and enter data in it It's it's a very easy system to learn Okay So now we're going to talk with just about a couple of other maybe more scientific focused Issues the first is I'm going to talk more about the importance of doing Studies that have adequate statistical power The as I go into this What is statistical power and why should we care about it? Most basically when we talk about power and statistics we mean The probability of detecting a difference which is considered to be of clinical interest or of clinical importance Okay, so another way of saying this a little more formally. It's the probability of being able to Detect a true positive result where by true positive result we mean a a relationship that is hypothesized Under the research hypothesis as being a clinically important relationship So if there is this true relationship out there, what's the probability that will actually have enough of a sample size to detect that? relationship that's what power is if power is high and You get a non-significant result where you don't get statistical significance That means you still have an informative study, right if the power is high And you get a null result a non-significant result that means you can rule out there being a an important relationship You can say there was not an important relationship between these variables and your findings should still be publishable negative findings are just as important as positive findings So this means that negative non-significant studies will be informative when power is high Studies with high power give researchers greater confidence that a significant result reflects the truth I'm going to come back to this in a second It turns out that if you get a positive result having done a low power Inadequately powered study. There's a very good chance that that's a false positive result And that's a very good chance that if you publish it in the literature that you've actually published something that's not true Now if you have low power and and do a And get a negative result, which is actually a very common situation Then you really don't have anything to say right you just simply say you did not achieve statistical significance But power was low so there that does in that situation There might have been an important effect But you just couldn't detect it you didn't have enough power to find it or there may have been no effect and and and then Consequence you got a negative result and you won't know which is which so basically in the setting of low power a Nonsignificant result is completely not informative. It just doesn't tell you anything If you are in a low power setting and get a positive result Maybe you get a hint of something going on there, but there's a very good chance It's a false positive. So so for these reasons we really emphasize adequate power for studies. Yeah Well, okay, we Yeah Yeah, well, you know, it's a that's a difficult thing if you see a trend In a low power situation you basically have it what what you know It could be what all you can really say to that is maybe another study should be done that has more power You can't really conclude that the trend you're seeing is real But you could just simply say well this we did a small sample size study and the trend is consistent with an effect Another we're gonna have to do more research to figure this out. I mean that that's pretty much what the conclusion is there You could think of an analog like like say you think you know since this is election season You haven't a pit say you do That you have an expectation that one candidate's ahead by 10 percentage points But you do it opinion poll that's got 10 people in it Right, and so you get you get a you do the the poll and you find and you end up concluding There's a 10 percent point 10 percentage point, you know the poll suggests a 10 percent difference That's consistent with there being That much difference in reality, but you really don't know the margin of error is so huge that it's just not very informative by itself But really all what you would the proper interpretation of that is as well that that is consistent with this 10 percent point difference But we're gonna need to do a bigger survey to really find out And that that's what it would and that's how I would characterize trends and underpowered studies Now this to go through this a little bit again Little formally, you know, how do we think of statistical power? This is a little simplified because I'm assuming a Dicotomous kind of scenario Whereas in reality in the real world we usually have continuum that we're dealing with but We can envision in this dichotomous scenario a two by two table where the Columns of the table represent whether or not the the There's a the intervention if we're evaluating an intervention Has no effect versus does it have a true effect? And the rows of the table have to do with the report our results, right? So the columns have to do with underlying truth What they what's really true in nature and the rows have to do with the results of our study, right? So our study can either report the results as negative or report the results as positive The and then there's four possibilities So if in truth there is no effect and we report results as negative then we got the right answer. That's correct The other another possibility is that there really is a true effect in nature In reality, but we've got negative results This failure to detect a true effect is referred to as a type 2 error And sometimes the probability of this type 2 error is denoted by the Greek letter beta So that's sometimes called the beta error Another possibility is in truth There's no effect in the reality, but we get a positive result and that's referred to as a type 1 error Usually denoted by the symbol alpha And then the other possibility is there really is a true effect And we detected it. We've got a positive result and the probability of that is what we mean by statistical power Okay, this is the situation where there really is an effect out there and we got a positive study that detected that effect Probability of doing that is power Now I've expressed this in terms of randomized trials ours, which is what I meant by RCT and So we're talking about effects of an intervention versus Are not the same ideas apply for any kind of relationship that you might be evaluating statistically So no effect you could replace that with no relationship and true effect could be replaced by The hypothesis that there really is a relationship in what you're studying Okay So what is statistical power power again is the chance of detecting a treatment effect, which is really there and Usually we try to set the power to be pretty high the night in the 80 to 95 percent range for a plausible effect size again We counterbalance power with the chance of a type one error Which is the probability of incorrectly rejecting the null hypothesis when the null hypothesis is actually true The type one error generally and statistic when we're doing research studies Should be kept to be less than five percent Sometimes smaller depending on the situation, but generally speaking it should be less than five percent We keep this probability low because it's important that as we accumulate a body of scientific research that we don't clutter the body of Reported positive findings with findings that are actually false right if we because we you know as we build the research As we build up knowledge in a certain area. It's important that that knowledge base be kept free of false findings So that's that's really one of the reasons why we keep the type one error low Now Study sample size power analysis a really part and parcel of the overall study design process And so when we're working out sample size and power this is very much an interactive process As we where we would start off with the research question specifying the design and major variables Then we come up with hypotheses you can think of these as reflecting infinite data sets So we don't have any kind of random noise or whatever But we're making hypotheses about the population we're studying or about the underlying reality that we're studying What effect size do we expect? And then we go on to formulate statistical methods and do a sample size calculation Very often when we do the sample size calculation We find out that it's not feasible to do this research, right? We find out that this that the study design is just not going to provide adequate power and when that happens we have to go back and Either refine the design refine the research question refine the outcome variables Change things around a little bit and we will keep this process going until we actually finally come up with a A sample size that's feasible within a study design to answer the research question something that's Should be emphasized is there really have been a number of arguments that doing underpowered studies It's not just bad research, but it's been this has been criticized as unethical if it's certainly if you're doing research involving patients and the argument here is that if you Sell a patient on doing a research study The patient is envisioning that they actually are contributing to to the acquisition of knowledge that they're contributing something to the world But if you have them doing an underpowered study in some ways the argument is is that you are misleading the patient that they're really not contributing to To a study that that could actually definitively answer a question I Want to just take another angle of this to sort of illustrate a little bit more the implications of having inadequately powered studies I talked about the type 1 and type 2 error flip side of that is When you after you publish the result of your study What is the probability that that result that you've published is incorrect? This is a little different than the type 1 and type 2 error The the implications of what you publish Can be expressed in terms of the positive conclusion error rate and the negative conclusion error rate positive conclusion error rate is once you've published the result and concluded that it's a positive finding What is the probability that that positive finding is in fact a true positive? and we're gonna That's the probability of a positive conclusion Given a true positive find a true positive I mean given that you've published a positive result and you can think of that as a positive predictive value associated with the result the corresponding error rate a positive conclusion error rate means that's the Probability that if you report a positive conclusion, what is the probability that that conclusion is an error, right? so and then correspondingly the negative conclusion error rate is What's the probability that if you report a negative conclusion that that negative conclusion is in an error Okay, and you can think of this you could attribute these error rates to the published literature When you find a positive conclusion reported in the literature What's the probability that that conclusion is an error that would be the positive conclusion error rate? And if you see a negative Conclusion reported in the literature, what's the probability that's an error? That would be the negative conclusion error rate These positive and negative conclusion error rates it turns out that they depend on power Is indicated by this formula, but they also depend on another quantity A ratio which is called sometimes just are in the literature and this is the ratio Of the number of true effects to the number of null effects that's tested by a research program And we can think of this ratio is reflecting The focus of the program reflecting the extent to which hypotheses that are tested Have really been vetted well thought through well Probability it's the ratio of hypotheses that are actually true Higher our values reflect well-focused research programs that by the time you get to statistical testing the hypotheses have a reasonable chance of being true low values of R a Research program with a low value of R can reflect for scattershot research where you're just sort of testing things willy-nilly So the positive predictive value It's the probability that if reporter if a finding is positive. What's the probability that it's really positive this is One minus the false discovery probability, or this is what I called on the preceding slide the positive conclusion error rate One minus the positive predictive values that positive conclusion error rate So if you have you want to have a high positive predictive value so that your positive conclusion error rate is low So I'm going to give you a couple examples And just sort of let's think through which result might be more impressive So let's suppose we have Sam the Sam shotgun one to well-focused They both have research programs going on but let's suppose that Sam shotguns research program He's just rapidly turning out things without really thinking things through very well And his R value is 10% which means that 10% of the research hypothesis evaluated in Sam shotguns research program are true But Wanda well-focused it's more thoughtful research The hypotheses are more refined and well developed and we'll suppose that 60% of the research hypothesis are true for Wanda well-focused Sam shotgun Obtains a p-value of 0.02 based on an underfunded study where the actual true power is just 30% Okay, the true power being quite low Wanda well-focused gets a p-value that's higher 0.04 based on a well-funded study with good statistical power of 90% So the even though the p-value is lower for the Sam shotgun study turns out that The research with that's the the higher p-value in the well-focused study actually does provide more compelling evidence And I'll try to explain this here This just represents a couple of scenarios here The True hypothesis rate I use theta here I apologize for that should have been R to be consistent with the last slide But with this R of either 10% or 60% And then we've assumed either the true statistical power is 30% or 90% and the Positive and negative conclusion error rates are given in the final two columns And so some of the things we can see here The and and I'm assuming throughout that the type one error is 5% so if we go to the Inadequately powered study with a low R. So unfocused research underpowered work 60% of the time Positive results these are results that have p-values less than 0.05 You know results that are nominally statistically significant at the 5% level 60% of that work which is reported as positive is actually an error right that means 60% false positive There's a 60% chance that any positive conclusion is actually a false positive conclusion way higher than that 5% alpha level There's also an 8% chance that if you get a negative result that that's a false negative in that case Going to a larger sample size in this kind of shotgun unfocused research approach Reduces the positive conclusion error rate some but it's still reasonably high Now if you have focused research If it's focused but underpowered You still have pretty high Positive conclusion and negative conclusion error rates certainly higher the positive conclusion error rates still higher than that 5% level that we feel comfortable with so you really need to have both Well-focused research to get that positive conclusion error rate down and an adequately large sample size So they keep to keep the positive conclusion error rate under 5% that's really the only way you can do it You'll you'll have With that kind of scenario you're still going to have some some risk of negative conclusions This is really just a trade-off that we make in science Yes Yeah, so the time So the type one error it does depend on a couple of things One issue is sometimes you reduce type one error rate in scenarios where you're doing multiple analyses so one way of If you can't be focused enough to have a single pre-specified primary analysis One way of dealing with that is re is reducing the threshold for type one error And that gives you protection In a global sense of making type one errors across multiple different analyses The other reason you might reduce that type one error threshold is if Is in settings where a false positive where the risk of a false positive conclusion is very high Where you want to avoid that? FDA before they approve a new labeling for a drug They usually focus on either having two studies with a type one error less than five percent Or if you're doing a single study that means if it's 0.05 times 0.05 so 0.00025 So they they they put a very low threshold Sometimes they negotiate this but but that's their the principle and the reason for this is is they really don't want to approve a Drug or product getting on the market if in fact it's not effective, right? So they they they will sometimes use very very low thresholds Oh Well, I you know, I think ideally you mean at least in my mind I think this is when we're thinking of the five percent alpha I think the same reasoning processes that apply to the five percent threshold apply to the positive conclusion error rate to So I would argue you really would like the positive conclusion error rate to be less than five You know, you would like that not to be greater than five percent usually These things are Context dependent So there might be circumstances where you could afford a higher one One place where you might afford a higher one is where the result of your study isn't going to be to affect clinical practice or even affect the Body of scientific knowledge, but it's just considered a preliminary study that's going to lead to another study later So in early phases of research when you're doing Pilot study preliminary study those might be settings or a higher positive conclusion error rate would be acceptable Yeah, yeah That's a very tough one and and and I I think to be honest for the most part This is a conceptual kind of Paradigm as opposed to one you used and day-to-day practice Because it really is very very difficult to determine a true R But it you can kind of get the conceptual flavor of well-focused versus scattershot research Now the CNEDs guy who I mentioned a minute ago who had published the article about most research findings Most medical research is false He's done and his he's got a whole team Investigative team at Stanford that they do all kinds of meta-analyses and systematic reviews and they've made arguments that Trying to evaluate this our factor based on You know published research It depends a lot on the context of the research But they have tried to come up with some vague ideas of what are is in different settings but by and large This is I should be clear there I'm bringing in this are more to make a point than to think of something that you actually bring into a research study Okay This is another graph That just gives a little more detail of that single example. I just gave the positive predictive value is on the vertical axis and again, you can think of One minus the prop positive predictive value is that probability of a false positive That's the positive conclusion error rate So you'd like a positive predictive value to be as high as possible And it would need to be point nine five in order for the positive conclusion error rate to be less than five percent But you can see here that are is very very important doing well focused research is very very important in order to Keep the positive predictive value high, but power is too and The the the challenge is is that if the sample size calculation the specifying of effect size is not being done Correctly not being done an unbiased way You can very often end up with true statistical power in this 10% to 50% or so range And if you combine that with a low R you really are in settings where the positive predictive value can be quite low Okay, I'm going to finish off by talking about another topic Which can compound challenges with low R or with low statistical power and and this is the issue of Selective reporting okay, and this is what happens when we don't do analysis plans and we just start Doing a bunch of analyses and then do a bunch of analyses and then we report Selectively a certain set of those analyses in publications Now I'm going to illustrate this this is I'm using a kind of acute example, but I think this kind of illustrates the idea this is actually something that was done by a tabloid London Times back in 2004 and Someone there or someone had collected the Zodiac birthsigns of 1067 rich people and They they noted That among these rich people Gemini birthsign occurred You know more often a hundred and ten times Compared to Pisces right so among the rich seemed like more of them were were Gemini than Pisces and You know if you go look up what Gemini and Pisces mean in like your astrological Tables or whatever you You get descriptions of Gemini along the lines of driving force of Gemini's is Conversation of the Gemini's conversation is their mind so they're very intellectually inclined probing people In search of information Pisces are more wishy-washy selfless spiritual Focus on their inner journey dreams and secrets Comfortable in an illusory world So you could Without too much difficulty if you go back if you're going back and looking at this result kind of make a story Right you can make a post-doc story to explain why Gemini's would be more likely to accumulate wealth than Pisces who are after all just living in an illusory world The and so and that the article would look at that look at that and they conclude well This is basically proof that astrology is valid right because we validly verified that in accordance with the descriptions of these signs that You know Gemini's were better at accumulating wealth Can anybody and in fact if you do a statistical test just straight comparison of the frequency of Gemini's to the frequency of Pisces and these wealthy people you get a p-value. That's low. It's 0.006 What's the problem with this There's a few problems actually But the yes the main problem that is the post-doc nature of this right you got 12 different zodiac signs So if you look at all comparisons among those 12 signs That's 12 times 11 over 2 that's 66 comparisons And so that's a lot of different chances to get a significant result. It's very unfocused. It's not pre-specified It would have been different if they before looking at these signs had said we're gonna test the hypothesis that Rich people are more likely to be Gemini than Pisces right. It's very different that if you try to tell the story afterwards you can I mean at least in my experience working with clinicians Even my myself I'll say I shouldn't blame clinicians for this just my experience in life is just about any see any Piece of data you can come up with you can find a story to justify that And if you find this story post-doc It's very very then easy to deceive yourself to think that that was pre-specified and that's sort of the process that goes on Now another issue you could bring in the issue of R You know that the fact we did you know how likely is the research hypothesis to be true And I think most of us would probably regard R is not being very high here Maybe not zero because we'll be open-minded, but it's probably not very high That zodiac signs really would determine well So oops, sorry, let me go back here So anyway, if you just dealing with the post-hoc issue We can do statistical adjustments to account for the fact that there really are 66 pairs of zodiac signs And one approach is using a Bonferroni adjustment and had we done that The p-value would have been point four one another thing we could have done is not just Post-hoc in post-hoc fashion isolated these two signs But just compare all 12 the frequencies across all 12 signs And if we did that the p-value would have been point two six right So if you do a proper p-value here Accounting for the unfocused nest nature of the hypothesis than the p-values are well above point oh five This slide Kind of illustrates the combination of all the what happens with all these bad practices put together So if you're doing scattershot research with low r-value post-hoc selection across multiple analyses and low power You end up in this situation over here Where your positive predictive value if you get a positive finding doing that kind of research Maybe like 25% or so whereas well-focused research good statistical power and Not doing post-hoc types of analyses your positive predictive values up there around 95% I didn't fully describe this The different curves in these plots are representing the number of Separate analyses that were done the number of separate independent analyses that were done Where we're assuming that the researcher is publishing The most promising of the different analyses the most positive The analysis with the lowest p-value across all those analyses So if any equal one that means you had you had an analysis plan that pre-specified a single primary analysis In equal eight means they did eight different analyses and they just reported the one that was the most consistent with the research hypothesis so again big difference between in the positive predictive value between high power good focus and no data dredging versus low power Poor focus and high data dredging Okay, so this is why we really harp on these kinds of issues and really harp on pre-specifying analyses And you know, I've got a few more slides. I'm going to actually skip over these If you're writing grants, I'm probably that's a later stage in research for most of you guys But we work a lot on grant writing Same actually the same ideas apply to Research in general But I just wanted to give a kind of some of the examples of some of the things we might do If early on it looks like the sample size that's feasible that you can actually attain isn't high enough There's actually many many many many different kinds of strategies that we can look at just from a statistical perspective to try to help you come up with a well-powered study And my last couple of slides here give some examples of that And I think you guys will have access to these. I know I think I sent them to the group so you can look through some of these Okay Any questions? Okay, well, let me just say again as you get ready for your research projects If you are going to work with our group just come soon That's the main the main message So thank you very much. I enjoyed talking with you all today