Small samples do not cause greater accuracy—but clear data may cause small samples: Comment on Fiedler and Kareev (2006).

Abstract

Fiedler and Kareev (2006) have claimed that taking a small sample of information (as opposed to a large one) can, in certain specific situations, lead to greater accuracy--beyond that gained by avoiding fatigue or overload. Specifically, they have argued that the propensity of small samples to provide more extreme evidence is sufficient to create an accuracy advantage in situations of high caution and uncertainty. However, a close examination of Fiedler and Kareev's experimental results does not reveal any strong reason to conclude that small samples can cause greater accuracy. We argue that the negative correlation between sample size and accuracy that they reported (found only for the second half of Experiment 1) is also consistent with mental fatigue and that their data in general are consistent with the causal structure opposite to the one they suggest: Rather than small samples causing clear data, early clear data may cause participants to stop sampling. More importantly, Experiment 2 provides unequivocal evidence that large samples result in greater accuracy; Fiedler and Kareev only found a small sample advantage here when they artificially reduced the data set. Finally, we examine the model that Fiedler and Kareev used; they surmised that decision makers operate with a fixed threshold independent of sample size. We discuss evidence for an alternative (better performing) model that incorporates a dynamic threshold that lowers with sample size. We conclude that there is no evidence currently to suggest that humans benefit from taking a small sample, other than as a tactic for avoiding fatigue, overload, and/or opportunity cost-that is, there is no accuracy advantage inherent to small samples.