The fields of psychology and cognitive neuroscience have had some rough sledding in recent years.
The bumps have come from high-profile fraudsters, concerns about findings that can’t be replicated, and criticism from within the scientific ranks about shoddy statistics. A new study adds to these woes, suggesting that a wide range of neuroscience studies lack the statistical power to back up their findings.
This problem isn’t just academic. The authors argue that there are real-world consequences, from wasting the lives of lab animals and squandering public funding on unreliable studies, to potentially stopping clinical trials with human patients prematurely (or not stopping them soon enough).
“This paper should help by revealing exactly how bad things have gotten,” said Hal Pashler, a psychologist at the University of California, San Diego. Pashler was not involved with the new study, but he and colleagues have previously raised concerns about statistical problems with fMRI brain scan studies in human subjects.
The aim of the new study wasn’t to rake neuroscientists over the coals, but to get them talking about how to change the culture and the incentives that promote statistically unreliable studies, says co-author Marcus Munafò, a psychologist at the University of Bristol, United Kingdom. “We’re really trying to be constructive about this.”
Statistical power is essentially the probability that a study will detect an effect of a given size if the effect is really there. It depends on two things: the sample size (the number of people in a study, for example) and the effect size (such as a difference in brain volume between healthy people and Alzheimer’s patients). The more people in the study and the bigger the size of the effect, the higher the statistical power.
Low statistical power is bad news. Underpowered studies are more likely to miss genuine effects, and as a group they’re more likely to include a higher proportion of false positives — that is, effects that reach statistical significance even though they are not real.