Abstract

AbstractMany large‐scale performance assessments include score resolution procedures for resolving discrepancies in rater judgments. The goal of score resolution is conceptually similar to person fit analyses: To identify students for whom observed scores may not accurately reflect their achievement. Previously, researchers have observed that rater‐agreement methods and person fit analyses result in similar conclusions about which students’ achievement estimates warrant additional investigation, and that score resolution generally improves person fit. We consider the implications of using person fit analysis as an initial step to identify performances for score resolution, and of using fit indices to identify raters to provide additional ratings. We simulated student responses to multiple‐choice items and a writing task. We simulated various types of person misfit in the writing task and identified the persons who needed resolution using a person‐fit index. Results indicate larger improvements in person fit after resolution when the person fit approach was used compared to a rater‐agreement approach. With the fit‐informed approach, person fit improved for of the misfitting students; with the rater‐agreement approach, person fit improved for around one third of these students. We consider the implications of our findings for mixed‐format assessments.

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call

Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.