Clarifying the advantage of small samples: As it relates to statistical Wisdom and Cahan's (2010) normative intuitions.

Abstract

On the basis of earlier findings, we (Fiedler & Kareev, 2006) presented a statistical decision model that explains the conditions under which small samples of information about choice alternatives inform more correct choices than large samples. Such a small-sample advantage (SSA) is predicted for choices, not estimations. It is contingent on high constant decision thresholds. The model was harshly criticized by Cahan (2010), who argued that the SSA disappears when the threshold decreases with increasing sample size and when the costs of incorrect decisions are higher than the benefits of correct decisions. We refute Cahan's critique, which confuses normative and descriptive arguments. He neither questioned our theoretical reasoning nor presented empirical counterevidence. Instead, he discarded our model as statistically invalid because the threshold does not decrease with increasing sample size. Contrary to this normative intuition, which presupposes a significance-testing rationale, we point out that decisions are often insensitive to sample size. We also refute Cahan's intuition that ignoring the potential asymmetry of gains and losses creates a serious bias in favor of the SSA. We regret any misunderstandings resulting from our linking the SSA to Bernoulli's law of large numbers.