Enter the concept of effect size. Effect size gives one a way to think about how large an effect (e.g. a height difference) is, not just the probability of the data given the null hypothesis.
Social science research puts the p-value on a pedestal. The p value, or probability of the data given the null hypothesis is true, is seen as the gateway to publication, giving authors an incentive to “p hack“, or use various tricks to get p-values down below .05. And they do this despite the lord loving the .06 as much as the .05. We have cartooned on this before: One gripe with the p-value is that statistical significance is cheap. Most plausible hypotheses become statistically significant when the sample size is large enough. Among other things, statistical significance is a function of sample size. In the age of mTurk-scale data, attaining statistical significance is easier than ever. We have heard it said that that if you draw a line anywhere through the belly of the United States, you’ll find a significant different in height on opposite sides of the line because of the massive sample size. But it may be a puny difference.
Enter the concept of effect size. Effect size gives one a way to think about the magnitude of effects, not just the probability of the data given the null hypothesis (aka, the p-value). One popular measure of effect size, Cohen’s D, is discussed along with in the beautiful visualization pictured above. Learn more from the article It’s The Effect Size, Stupid, from which this post gets its name. And learn why you need lots of data to estimate effect sizesfrom our friends at Data Colada.