Is the List Experiment Doing its Job?

Abstract

This paper sheds new light on the unobtrusive measure known as the list experiment or unmatched count technique. Proponents of this method claim that it detects social desirability bias in responses to sensitive questions in surveys. The logic of this method is quite straightforward. After a critical overview of the theory, logic, and empirical results of this type of measure, we present the results of a series of three studies. While the first study yielded promising results, the replication of the outcome pattern in Study 2 failed completely. The third study, based on longitudinal data, delivers indications for the systematic inconsistency of the claimed logic of the list experiment. First, significant mean differences between baseline and test condition occur even if the additional item is non-sensitive and has an agreement rate of about two percent in direct questioning. Second, test-retest reliability shows fluctuating results depending on the sensitivity and number of items included in the experiment. Implications for theory and practice in measuring social desirability by unobtrusive measures are discussed.