Shifting Conceptions of Validity in Educational Measurement: Implications for Performance Assessment

Abstract

Recent developments in the philosophy of validity, highlighting the importance of investigating the consequences of assessment use, provide theoretical support for the move toward performance assessment. The problem for validity researchers is finding the appropriate set of criteria and standards to simultaneously support the validity of an assessment-based interpretation and the validity of its impact on the educational system. My intent is to provide an integrative and critical review of the guidance available for conducting validity inquiry in the context of performance assessment. In the first section is a summary of the emerging consensus among measurement scholars—not reflected in the 1985 Standards for Educational and Psychological Testing (American Educational Research Association [AERA], American Psychological Association [APA], & National Council on Measurement in Education [NCME])—about the centrality of construct validity to the evaluation of any assessment-based interpretation and about the importance of expanding the concept of validity to include explicit consideration of the consequences of assessment use. The description of this emerging consensus suggests general epistemological principles for validity inquiry. In the second section is a description and synthesis of various categories of questions, evidence, or criteria that have been used to describe or guide validity inquiry, either for assessment in general or for performance assessment in particular. These analytic schemes, like the traditional construct-content-criterion categories, highlight specific issues that their authors consider important for validity researchers to address. Each balances technical concerns about such issues as reliability, generalizability, and comparability with concerns about the consequences of assessment. In the final section is an overview of concerns, expressed largely by interpretive researchers, about validity criteria that privilege standardized forms of assessment, whether performance-based or multiple-choice. These arguments suggest the importance of further expanding the conception of validity inquiry to treat as problematic the epistemological principles used to warrant validity conclusions.