Developing a Machine‐Supported Coding System for Constructed‐Response Items in PISA

Abstract

Approximately a third of the Programme for International Student Assessment (PISA) items in the core domains (math, reading, and science) are constructed‐response items and require human coding (scoring). This process is time‐consuming, expensive, and prone to error as often (a) humans code inconsistently, and (b) coding reliability in international large‐scale assessments varies across items and countries. The shift in PISA 2015 from paper‐ to computer‐based assessment digitized all responses and associated coding, thus providing opportunities to introduce technology and analytical methods to improve data processing and analyses in future cycles. In this computer‐based setting, this research explains the framework and approach for improving the accuracy and efficiency of the coding process in constructed‐response items for future PISA cycles. In the pilot study associated with this research, we demonstrate how the proposed machine‐supported coding system can be utilized using sample reading items from the PISA 2015 cycle.