Investigating rater accuracy in the context of secondary-level solo instrumental music performance

Abstract

In any performance-based musical assessment context, construct-irrelevant variability attributed to raters is a cause of concern when constructing a validity argument. Therefore, evidence of rater quality is a necessary criterion for psychometrically sound (i.e., valid, reliable, and fair) rater-mediated music performance assessments. Rater accuracy is a type of rater quality index that measures the distance between raters’ operational ratings and an expert’s criterion ratings on a set of benchmark, exemplar, or anchor musical performances. The purpose of this study was to examine the quality of ratings in the context of a secondary-level solo music performance assessment using a Multifaceted Rasch Rater Accuracy (MFR-RA) measurement model. This study was guided by the following research questions: (a) overall, how accurate were the rater judgments in the assessment context? (b) how accurate were the rater judgments across each of the items of the rubric?, and (c) how accurate were the rater judgments across each of the domains of the rubric? Results indicated that accuracy scores generally matched the expectations of the MFR-RA model, with rater locations higher than the average student performance, item, and domain locations, indicating that the student performances, items, and domains were relatively easy to rate accurately for the sample of raters examined in this study. Overall, rater accuracy ranged from 0.54 logits ( SE = 0.05) for the most accurate rater to 0.24 logits ( SE = 0.04) for the least accurate rater. Difficulty of rater accuracy across items indicated a range of 0.91 logits ( SE = 0.08) to -1.83 logits ( SE = 0.17). Difficulty of rater accuracy across domains ranged from 0.25 logits ( SE = 0.08) to -0.68 logits ( SE = 0.17). Implications for the improvement of music performance assessments with specific regard to rater training are discussed.