Evidence to support test interpretation: Evaluating the content validity of a new item set for the Transition Pragmatics Interview

Abstract

Purpose There is little consensus on evidence-based practice guidelines for the selection of criterion-referenced assessments. Having confidence in scores from criterion-referenced assessments requires evidence that items align with their intended constructs. The purposes of these studies were to demonstrate evidence of content validity for the revised item set of a developing social communication assessment and to provide clinicians with a model of content validity evaluations that can be generalised to the review of other assessments. Method In Study 1, 10 experts rated 25 newly-developed items for how well they represented the intended construct. In Study 2, seven participants ages 14–20 were administered the Three Step Test Interview to assess their cognitive processes for responding to new items. Examinee responses were coded for construct-relevant and construct-irrelevant factors. Result Twenty-three of the 25 newly-developed items were deemed representative of the intended construct by experts and elicited construct-relevant response processes from examinees. Conclusion The integration of expert review and examinee cognitive interviewing provides a more complete evaluation of the alignment of the items to their intended construct. Transparent reports of the methods and findings of content validity studies strengthen the ability of clinicians to select criterion-referenced assessments that support valid decisions.