The Performance of Vignettes in Focus Groups and Cognitive Interviews in a Cross-Cultural Context

Abstract

Survey researchers conducting pretesting via cognitive interviews or focus groups often use vignettes to evaluate questions and answer categories in order to identify possible measurement problems, particularly for reporting situations that are relatively rare or for sensitive topics. Although research has compared the performance of vignettes in Spanish and Asian languages in cognitive interviews, there is little research that compares the performance of vignettes across pretesting methodologies in languages other than English. To address this gap, we investigated the performance of a vignette about a homeownership question that was administered in focus groups and cognitive interviews conducted in seven languages: English, Spanish, Chinese, Korean, Vietnamese, Russian, and Arabic. We coded the cognitive interviews and focus group summaries to quantify the responses and the types of comprehension problems respondents had, and we compared the vignette response data from cognitive interviews to the data from focus groups, by language. We find that administering the vignette in cognitive interviews was more effective than administering the vignette in focus groups for uncovering difficulties respondents had with the survey question, particularly for Spanish- and Arabic-speakers. We conclude that using vignettes in focus groups without cognitive interviews may not reveal problems with survey questions as effectively. Regardless of methodology, the vignette task was challenging for certain language groups, and further research is needed on the cross-cultural adaptation of vignettes.