Participant recruitment methods and statistical reasoning performance

Abstract

Optimal Bayesian reasoning performance has reportedly been elusive, and a variety of explanations have been suggested for this situation. In a series of experiments, it is demonstrated that these difficulties with replication can be accounted for by differences in participant-sampling methodologies. Specifically, the best performances are obtained with students from top-tier, national universities who were paid for their participation. Performance drops significantly as these conditions are altered regarding inducements (e.g., using unpaid participants) or participant source (e.g., using participants from a second-tier, regional university). Honours-programme undergraduates do better than regular undergraduates within the same university, paid participation creates superior performance, and top-tier university students do better than students from lower ranked universities. Pictorial representations (supplementing problem text) usually have a slight facilitative effect across these participant manipulations. These results indicate that studies should take account of these methodological details and focus more on relative levels of performance rather than absolute performance.