Underestimating the instruction: a meta-analysis on item count technique and crosswise model

Abstract

Survey respondents tend to present themselves in a more favorable light, especially when being asked unpleasant questions. This so-called social desirability bias introduced by sensitive questions often distorts survey responses. As a remedy research draws on indirect questioning formats that aim to protect respondents’ privacy and ensure their anonymity. Two prominent examples of such techniques are the Crosswise Model (CM) and the Item Count Technique (ICT). Both methods follow unconventional structures using group answers or known distributions to mask individual answer but that also require long, complex and dense instructions. Previous research has suggested that ICT and CM produce more truthful answers, however they impose a higher cognitive burden on respondents. Although, it is commonly believed that respondents fully understand and follow these more demanding instructions, recent research suggests that this is not always the case. To further investigate this notion, I conduct a meta-analysis of the ICT and CM and analyze the instructions of these methods to answer two core questions: First, how do the implementations of the Item Count Technique and the Crosswise Model differ across studies? Second, how do specific characteristics (i.e., the instruction) of the techniques affect their performance? The meta-analysis indicates mixed results on the performance of the techniques. The CM tends to perform better than the ICT. ICT works best when asked in face-to-face interviews, the sensitive item phrased as a socially undesirable one, and the non-sensitive items chosen from the same contextual background. ICT instructions with too many words and not many word repetitions appear to have a negative influence on its outcome. The results of this research have implications for researchers and practitioners working with these techniques, but also for the broader field measuring and analyzing sensitive characteristics in surveys. Keywords: Item Count Technique; Crosswise Model; Meta-Analysis; Cognitive Burden; Survey Methodology; Instructions