Why Most Published Research Findings Are False

Dedicated to Tandykane, for being the umpteenth science denialist to throw this article at me: may you learn something from it.

John Ioaniddis' article "Why Most Published Research Findings Are False"
http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124

More about John.
http://en.wikipedia.org/wiki/John_P._A._Ioannidis

Alex Tabarrok on Marginal Revolution, a great article summarizing the math:
http://marginalrevolution.com/marginalrevolution/2005/09/why_most_publis.html

Orac's blog, Respectful Insolence, covers the misuse of this research in popular media and the Internet.
http://scienceblogs.com/insolence/2007/09/the_cranks_pile_on_john_ioannidis_w...

Here's Newsweek completely misinterpreting the finding to satisfy an anti-science narrative:
http://www.newsweek.com/2011/01/23/why-almost-everything-you-hear-about-medic...

Dr. Steven Novella on his blog Neurologica:
http://theness.com/neurologicablog/index.php/are-most-medical-studies-wrong/

Category:

Education

Tags:

• Medical
• research
• statistical
• hypothesis
• testing
• observational
• studies
• science
• statistics
• lies
• bias
• distortion
• licorice
• c0nc0rdance

[bsrk7]

I'd like to add that, according to my professor, a lot of the time the researchers don't tweak their datasets to be dishonest of deceitful, but more because they think they must have screwed up (since it doesn't correspond to other researchers' datasets) and, out of a lack of confidence (or to save face), they alter their data. They're afraid of making mistakes, basically. This kind of behavior and fear of being "wrong" needs to be purged from all scientific disciplines.

[C0nc0rdance]

@bsrk7

Your professor is very right. A lot of time the people making the tweaks are young grad students who want to be assured of not being embarrassed by having made a mistake. A lot of research is done by these young folks who were told by their boss what they should expect to get.

That's something we really should address as a scientific community, and I'm very grateful to Dr. Ioannidis for his insights on this topic. I know I learned something from his work.

[RationalWaves]

I've been doing some reading about probability, statistics, and p-values in particular and I've become increasingly skeptical of the way Fisher's p-value is widely interpreted. Google "P Values are not Error Probabilities" and "A Dirty Dozen: Twelve P-Value Misconceptions" for a couple papers that I found interesting.

In particular, I think you might be very wrong to equate the p-value with the false positive rate. From what I've read, for a p-value of 0.05, the type I error can be much higher.

[C0nc0rdance]

@RationalWaves

Yes, the Ioannidis article does a great job of explaining that to us "math-idiot" biologists. The ratio of false to true hypotheses, if it is very high, gives us an almost zero chance of a true positive, even at p<0.001

I suspect that is why surprising results come out of statistically based tests of parapsychology, like the "Global Conciousness Project". The ratio of false to true hypotheses there is probably approaching infinity.

[cly2tw]

Very dubious.

1. If Bayesian be the way to go and to judge scientific results by, is there any single research claim that is "correct" at the time of its publication accordingly, or in eyes of this self-claimed SBM people?

2. What is the right "prior" to start with? "Extraordinary claim", say acupuncture, is likely based on uncommon models. Why should it be subjected to judgment of 'competing' theories per se that makes it start with a unfavorable prior, rather than a fair clean-slate status?

[C0nc0rdance]

@cly2tw

PPV (positive predictive value) is based on prior work on the topic, the type of study, the plausibility of the underlying mechanism, the strength of the evidence, the level of significance, and an estimate of R, the ratio of "true-" to "no relationships" that could be included. These can be somewhat subjective, but not arbitrary.

Acupuncture trials have a near-zero PPV.

All of this is discussed in great detail in the paper cited and linked. Please read it.

[C0nc0rdance]

@MonotonePeanut

Good point, and I would agree in a hypothetical perfect world, but in practical meta-studies, if two studies are done at different places or times or by different people, there is no way to perfectly replicate even a simple outcome like weight gain or maze-running times. These variables are called replication confounders or replication artifacts.

The only solution is very large single populations, or meta-studies that treat each study as sampling from different populations.

[C0nc0rdance]

@PainefulMass

All my favorite blogs were covering it, and it just reminded me that I planned on making a vid about it. Ioannidis does very important work, I'm saddened that it's abused for anti-science arguments.

[msginca]

I don't know about anyone else, but I'm quite certain black licorice turns rats into unicorns.

[bsrk7]

Good video! But I object to your statement that black licorice tastes awful :P

I think that maybe we shouldn't dismiss the rare, contradictory studies so quickly. One of my professors, a toxicologist, was discussing that, in her experience, a terrifying # of researchers tend to tweak their datasets (only minor changes, but still) in order to make them agree with those of other researchers or to not "rock the boat". Any chorus of studies that agree should not be given a free pass from scrutiny.

[IceAges14Aces]

@C0nc0rdance You accused the "Global Consciousness Project" as biased due to data-dredging (which had been disproved) and now you're critcizing the GCP as just mere statistical artifacts? I agree that deviations away from the normal distribution are just chance in the long run, but this prob. is based on the significance level, not P. Since the significance level (Type I prob.) is too small in GCP, your criticism is again, invalid...

[jjrouse]

Thank you for this video! I'm taking Biostatistics right now and you have given a very helpful and intuitive review of frequentist hypothesis testing.