An attempt to decrease social desirability bias: The effect of cheap talk mitigation on internal and external validity of discrete choice experiments

Abstract

Discrete choice experiments (DCEs) have been used extensively to elicit preferences. However, the hypothetical nature of choices induces socially desirable behaviour and endangers internal and external validity of DCEs. This studyexperimentally investigated social desirability bias in DCEs and whether it can be mitigated using the cheap talk mitigation method in the context of food choices. Respondents (N = 1027) were randomly allocated to one of four questionnaire versions: default without manipulation, priming socially desirable behaviour, cheap talk mitigation, or both. The effect on internal validity was assessed by comparing respondent-reported characteristics, DCE results, and prediction accuracy for a holdout task between questionnaire versions. The effect on external validity was assessed by comparing stated and revealed preferences. Social desirability bias, if present, was hardly affected by cheap talk mitigation. Respondent-reported characteristics, DCE results and prediction accuracy for the holdout task and actual food choice did not strongly differ between questionnaire versions.Prediction accuracy for the holdout task was lowest in the default version. Prediction accuracy for actual food choice was slightly better among respondents in the versions that were exposed to cheap talk mitigation. Social desirability bias was hard to detect and mitigate in this study, potentially due to limited social desirability, the effectiveness of the cheap talk mitigation method, and other sources of hypothetical bias. The differences in prediction accuracy indicates that cheap talk mitigation slightly improved external validity at minimum cost to internal validity. Recommendations for future research are provided.