"Why Most Published Research Findings Are False"
From the same guy who brought us a study showing that one-third of clinical studies are later contradicted by larger studies (excellent summary by Orac here), we now have a rather alarmingly titled paper arguing that more than 50% of all scientific published papers are wrong. New Scientist summarizes:
Most published scientific research papers are wrong, according to a new analysis. Assuming that the new paper is itself correct, problems with experimental and statistical methods mean that there is less than a 50% chance that the results of any randomly chosen scientific paper are true.Ironically, the award system in science means that the first person gets all the credit, while there's no glory in repeating the result, and only a little more in refuting it. Inevitably, this means that "hot" fields with many competing labs are particularly susceptible to false published results, as there is both intense pressure to publish first (no one wants to be scooped!) and a strong bias for pursuing (and publishing) only positive results. This has been true, most notably, of recent research in stem cells.
John Ioannidis, an epidemiologist at the University of Ioannina School of Medicine in Greece, says that small sample sizes, poor study design, researcher bias, and selective reporting and other problems combine to make most research findings false. But even large, well-designed studies are not always right, meaning that scientists and the public have to be wary of reported findings.
"We should accept that most research findings will be refuted. Some will be replicated and validated. The replication process is more important than the first discovery," Ioannidis says.
Eventually, things do get sorted out; results must be repeated at some point (if for no other reason than that you often need to build on previous protocols to get to the next step in a research program). But all of this means that one should take new results with a healthy dose of skepticism.
This news will be a two-edged sword in terms of psychological effects on practicing scientists: on the one hand, obviously it's depressing that if you discover something, odds are it will turn out to be false. On the other hand, you can probably still get a paper out of it even so.
[Via Crumb Trail.]
Update, 2 September: Alex Tabarrok has a nice clear explanation of why, statistically speaking, one would expect most research findings to be false.