Evaluating the Impact of Rater Effects on Behavior Rating Scale Score Validity and Utility

Abstract

Behavior rating scales represent one of the most commonly used types of assessments in school psychology. Yet, they suffer from a fundamental limitation: They are an indirect methodology influenced partially by student behavior and partially by rater perspectives. Thus, the current study utilized advanced analytic approaches to evaluate rater effects on the Academic Competence Evaluation Scales–Short Form–Teacher (ACES-SF-T) with a partially crossed sample of 132 fourth- and fifth-grade students rated by seven teachers. Results indicated that rater effects had a minimal impact on the predictive validity of ACES-SF-T scores for state achievement tests, but at the individual level, rater effects could lead to starkly different conclusions about students’ academic, social, and behavioral functioning. Implications for research and practice are discussed.