Exploring the Impacts of Different Score Resolution Procedures on Person Fit and Estimated Achievement in Rater-Mediated Assessments

Abstract

ABSTRACT Scoring procedures for many rater-mediated performance assessments include score resolution procedures in which a third rater adjudicates discrepancies between two raters’ ratings of the same performance. There are numerous approaches for calculating resolved scores that involve different combinations of the original and third ratings. Using data from two content-area writing assessments, we examined the impact of applying no resolution and applying three resolution techniques: TQ, Parity, and the Operational resolution method used in the assessment program on the values and psychometric quality of student achievement estimates. We observed substantial differences in person fit, a model-based measure of the psychometric quality of student achievement estimates, across methods. Our results highlight the importance of examining person fit as part of evaluating the psychometric quality of rater-mediated performance assessments in which score resolution procedures are employed.