Improving the Generalizability of Behavioral Science by Using Reality Checks: A Tool for Assessing Heterogeneity in Participants’ Consumership of Study Stimuli

Abstract

In attempting to draw bigger conclusions, researchers in psychology open their labs to more diverse groups of people. Yet even the most far-reaching theories must be tested with specific stimuli, materials, and methodology. To the extent that a study’s stimuli are familiar beyond the lab to groups of people writ large, an experiment is said to have mundane realism—a type of external validity. We propose that an experiment’s stimuli will vary in their relevance to each individual participant (such as how much they consume the stimuli outside the lab) and can be assessed using a tool: reality checks. We found that accounting for a study’s mundane realism, at the individual level, significantly altered a study’s results—which we found to be the case in testing well-established findings in psychology and behavioral economics. Our work suggests that measuring mundane realism (in addition to creating it) is a useful way of testing effects in psychology among the participants for whom the studies’ scenarios and decisions will matter most outside of the lab.