Scientific apophenia in strategic management research: Significance tests & mistaken inference

Abstract

Research summary: This article uses distributional matching and posterior predictive checks to estimate the extent of false and inflated findings in empirical research on strategic management. Based on a sample of 300 papers in top outlets for research on strategic management, we estimate that if each study were repeated, 24–40 percent of significant coefficients would become insignificant at the five percent level. Our best guess is that for about half of these, the true coefficient is very close to 0. The remaining coefficients are likely directionally correct but inflated in magnitude. We offer several practical individual and field level suggestions for reducing scientific apophenia, that is, our tendency to find and publish evidence of order where none exists.Managerial summary: This article analyzes the degree to which statistics in research on strategic management provide meaningful evidence for decision-making. Based on a sample of 300 papers, we estimate that 24%–40% of reported results would probably not be confirmed if the study were repeated. Our best guess is that about half of the reported results are wrong (B∼ = 0) and the other half of results too weak to find repeatedly. We conclude that scientific apophenia—the tendency to find evidence of order where none exists—is a serious problem in the literature on strategic management. We recommend replication of empirical studies to insure that they provide evidence for guiding managers. We also provide guidance for avoiding scientific apophenia in empirical research. Copyright © 2015 John Wiley & Sons, Ltd.