Resolving and Re-Scoring Constructed Response Items in Mixed-Format Assessments: An Exploration of Three Approaches

Abstract

ABSTRACT We explored three approaches to resolving or re-scoring constructed-response items in mixed-format assessments: rater agreement, person fit, and targeted double scoring (TDS). We used a simulation study to consider how the three approaches impact the psychometric properties of student achievement estimates, with an emphasis on person fit. We found that although TDS is not specifically designed to identify discrepant or misfitting score profiles, this approach can improve person fit for students who are also within a critical range of achievement. Our findings also suggest that TDS alone is not sufficient to provide a comprehensive picture of measurement quality because it does not directly consider person fit. We suggest that researchers and practitioners consider combining TDS with other approaches to score resolution. We consider the implications of our findings for research and practice.