Sweeping recommendations regarding effect size and sample size can miss important nuances: A comment on “A comprehensive review of reporting practices in psychological journals”

Abstract

Statistical significance tests done with low statistical power levels can result in reports of exaggerated effect sizes. Funnel graphs can show these exaggerations for a given area of research by revealing a negative correlation between a study’s sample size on the y-axis and the effect size obtain by the study on the x-axis. A recent paper by Fritz, Scherndl, and Kühberger (2013) recommended that effect sizes should not be reported when there is a negative correlation between sample size and effect size for a given research area. This recommendation fails to consider magnitudes of the negative correlations and thereby misses the opportunity to mitigate effect size exaggerations and approximate more correct effect size estimates. This comment explains both the incorrectness of the recommendation and the approach and calculations necessary to correct effect size estimates.