Rater Agreement on IQ and Achievement Tests: Effect on Evaluations of Learning Disabilities

Abstract

Protocols from 110 evaluations utilizing the Wechsler Intelligence Scale for Children-Third Edition (WISC-III) and the Woodcock/Johnson Tests of Achievement-Revised (W/J-R) were scored by two different raters to determine (a) whether subtests with more difficult levels of scoring yield lower interrater correlation coefficients, (b) whether scoring errors on subtests affect broad score estimates, (c) the effect of expertise of rater on scoring errors, and (d) whether scoring errors affect a learning disability determination based on IQ/achievement discrepancy. Scoring errors were found on almost 25% of Comprehension and Vocabulary subtests; however, the effect of these scoring errors was minimal. About 42% of Writing Samples subtests had scoring errors, resulting in a mean change of 1.75 points on the Broad Written Language Cluster subtest. On the WISC-III, but not the W/J-R, there were significantly more errors made by inexperienced testers. Scoring errors resulted in two cases in which learning disability determination would be changed. Overall, the study corroborates previous findings of strong interrater reliability on most subtests of common IQ and achievement tests and indicates that novice scorers are not likely to make scoring mistakes that will significantly impact an IQ/achievement discrepancy-based documentation of learning disability.