Applying Mokken Scaling Techniques to Incomplete Rating Designs for Educational Performance Assessments

Abstract

ABSTRACTA major challenge in the widespread application of Mokken scale analysis (MSA) to educational performance assessments is the requirement of complete data, where every rater rates every student. In this study, simulated and real data are used to demonstrate a method by which researchers and practitioners can apply MSA to incomplete rating designs. The results suggested that when rating data are collected such that each rater rates test-takers in common with at least two other raters, it is possible to use MSA techniques to evaluate rating quality in rater-mediated assessments with incomplete ratings. These results imply that it is possible to evaluate fundamental measurement properties in rater-mediated assessments using a nonparametric approach. Such analyses are important for evaluating the validity, reliability, and fairness of rater-mediated assessments.