Generalizability of Cognitive Interview‐Based Measures Across Cultural Groups

Abstract

We addressed the challenge of scoring cognitive interviews in research involving multiple cultural groups. We interviewed 123 fourth‐ and fifth‐grade students from three cultural groups to probe how they related a mathematics item to their personal lives. Item meaningfulness—the tendency of students to relate the content and/or context of an item to activities in which they are actors—was scored from interview transcriptions with a procedure similar to the scoring of constructed‐response tasks. Generalizability theory analyses revealed a small amount of score variation due to the main and interaction effect of rater but a sizeable magnitude of measurement error due to the interaction of person and question (context). Students from different groups tended to draw on different sets of contexts of their personal lives to make sense of the item. In spite of individual and potential cultural communication style differences, cognitive interviews can be reliably scored by well‐trained raters with the same kind of rigor used in the scoring of constructed‐response tasks. However, to make valid generalizations of cognitive interview‐based measures, a considerable number of interview questions may be needed. Information obtained with cognitive interviews for a given cultural group may not be generalizable to other groups.