Examining Evidence for the Validity of PISA Learning Strategy Scales Based on Student Response Processes

Abstract

The aim of this study was to investigate response-process based evidence for the validity of the Programme for International Student Assessment's (PISA) self-report questionnaire scales as measures of specific psychological constructs, with a focus on scales meant to measure inclination toward specific learning strategies. Cognitive interviews (N = 22) and descriptive and statistical analysis of Norwegian questionnaire data (N = 4279) provide converging evidence that (a) a non-trivial portion of students may not be providing responses to items that can be considered valid, for a variety of reasons; (b) students who perform more poorly on PISA's science scales are less likely to provide valid responses to the questionnaire items; and (c) quantitative examinations of response patterns provide limited ways of identifying students who may not be providing valid responses, although changes in the questionnaire design could strengthen the utility of these methods. These findings are discussed in the context of the larger interpretive argument surrounding the PISA learning strategy scales, and implications for future research on the measurement of learning strategies are discussed.