Addressing Cultural Context in the Development of Performance-Based Assessments and Computer-Adaptive Testing: Preliminary Validity Considerations

Abstract

Performance-based and computer-adaptive assessments have been proposed for use as instructional assessments and for selection, placement, certification, licensing, and other accountability purposes. As with other assessments, it is important that the development, administration, scoring, and reporting of these assessments, as well as the cultural context in which they are viewed by individuals and subgroups, be considered as much as possible within a construct validity framework. This article examines the research and the steps required to address these concerns. It supports the development ofa new conceptual framework and more explicit guidelines for designing culturally responsive assessments. The introduction of performance-based assessments (PBAs) and computer-adaptive tests (CATs) for large-scale accountability purposes has had a positive and revitalizing impact on the measurement community. These innovations have led to both a reexamination of the bases of test construction and evaluation as well as a rich expansion of the measurement models, theories, and beliefs underlying test development. Though much work will be needed in the coming decades to realize this expansion-for example, fairness issues related to the new assessments have yet to be addressed in systematic fashion (Boodoo, 1992)-the efforts of psychometricians such as Messick (1992), Snow (1993), and others provide firm beginnings (Boodoo, 1994). Additionally, the development, administration, scoring, and reporting of information derived from PBAs and CATs must be carried out such that the cultural context of individuals and subgroups is increasingly taken into account. A new conceptual framework is needed to carry out this complex and important task. To understand the impact of PBAs and CATs on the assessment of minority students, it is first important to view the development of these instruments in light of current research on and standards for addressing the cultural context of testing as a whole (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 1995; Educational Testing Service, 1987). Beyond that, a framework for the development of culturally responsive PBAs and CATs must be created. STANDARDS AND GUIDELINES FOR ADDRESSING CULTURAL CONTEXT IN TESTING In 1985, the American Educational Research Association (AERA), the American Psychological Association (APA), and the National Council on Measurement in Education (NCME) jointly published their Standards for Educational and Psychological Testing. This comprehensive document differentiates these standards into three categories: primary, secondary, and conditional. Primary standards are those that should be met by all tests before their operational use and in all test uses unless a sound professional reason is available to show why it is not necessary or technically feasible to do so in a particular case. Secondary standards are desirable as goals but are likely to be beyond reasonable expectations in many situations. Conditional standards are those that may be primary in some situations and secondary in others. The AERA/APA/NCME publication outlines seven technical standards that directly address cultural context in test development relevant to validity, reliability, test design, test content, test format, and norming: (1) Standard 1.20 (Conditional): Investigations of criterion-related validity for tests used in selection decisions should include, where feasible, a study of the magnitude of predictive bias due to differential prediction for those groups for which previous research has established a substantial prior probability of differential prediction for the particular kind of test in question. (AERA, APA, & NCME, 1985, p. 17) (2) Standard 1.21 (Primary): When studies of differential prediction are conducted, the reports should include regression equations (or an appropriate equivalent) computed separately for each group, job, or treatment under consideration or an analysis in which the group, job, or treatment variables are entered as moderators. …