The accuracy of performance task scores after resolution of rater disagreement: A Monte Carlo study

Abstract

► We used Monte Carlo methods to study the accuracy of reported scores from performance assessments. ► Four rater disagreement resolution models were studied. ► The study included both 4- and 6-point scoring rubrics. ► Scale length, rater severity, and proficiency of the expert and rater were studied in the model. ► The expert and parity methods of disagreement resolution were found to hold greatest promise. When multiple raters score a writing sample, on occasion they will award discrepant scores. To report a single score to the examinee, some method of resolving those differences must be applied to the ratings before an operational score can be reported. Several forms of resolving score discrepancies have been described in the literature. Initial studies of the various methods, however, have demonstrated that decisions about student performance may differ depending on the resolution method applied. Thus, studies are needed to investigate the quality of the scores associated with each model. To study score quality associated with each model, we conducted a Monte Carlo study and varied the factors associated with scoring and resolution to determine the conditions under which a particular resolution method might be superior.