How Cognitive Interviewing can Provide Validity Evidence of the Response Processes to Scale Items

Abstract

The current theory about validity reflected in the Standards for Educational and Psychological Testing (AERA et al. in Standards for educational and psychological testing, American Psychological Association, Washington, DC, 1999), offers no clear indications about the methods for gathering validity evidence about the response processes. Cognitive interviewing (CI) can play an important role answering the current demand about empirical and theoretical analyses of the response processes as a source of validity evidence in psychological testing. CI can provide validity evidence for investigating substantive aspects of construct validity and for contributing to the explanations for item and test scores (Zumbo in Handbook of statistics, vol 26, Elsevier, Amsterdam, pp. 45–79, 2007; The concept of validity: revisions, new directions and applications, IAP—Information Age Publishing Inc., Charlotte, NC, pp. 65–82, 2009). The aim of the study was to illustrate the use of cognitive interviewing method for gathering validity evidence on response processes. The search for evidence about the “response process” was guided by an argument-based approach to validity (Kane in Psychological Bulletin 1992; Educational measurement, American Council on Education/Praeger, Washington, DC, pp. 17–64, 2006). 21 cognitive interviews were carried out during the cognitive testing of the APGAR psychological scale intended to measure the “family support” construct. Cognitive interviewing provided validity evidence that explains how respondents interpret and respond to the APGAR items. Respondents maintained a shared interpretation of “family concept” while answering the APGAR scale items. Nevertheless, they included in the concept of family not only family members they live with but also other family members and even friends. CI participants were also capable of classifying their answers about the family support perception following a polythomous response system. Lastly, the role of CI in the Kane’s argument-based approach and Zumbo’s contextualized view of validity will be discussed.