Using virtual presence and survey instructions to minimize careless responding on Internet-based surveys

Abstract

Internet-based survey data inform knowledge creation in research and justify work activities in organizations. While there are advantages to online surveys, this mode of administration comes with its own set of challenges. Survey respondents may engage in careless responding (i.e. insufficient effort responding or satisficing) by intentionally or unintentionally responding in a manner that does not accurately reflect their true sentiments. Careless responding can create psychometric problems even after correctly removing careless respondents (i.e. mischievous responders). This study aimed to improve survey methodology by preventing careless responding. Using a 3×3 between-subjects experimental design, we manipulated both virtual presence (none, animated shape, and virtual human) and type of instructions (anonymous, warning, and feedback). Indicators of careless responding were the dependent variables. Results showed that beyond characteristics of survey items, survey design elements can prevent careless responding. The effects of interventions differed by type of careless responding. Instructions, and the interaction of instructions and virtual presence significantly reduced careless responding, but not virtual presence alone. Virtual human presence increased the salience of instructions. Although currently effective, instructions warning of punitive consequences may create difficulty in recruiting participants. Future research should continue investigating non-aversive ways to prevent careless responding on Internet-based surveys.