Computational Cognitive Modeling of Human Calibration and Validity Response Scoring for the Graduate Record Examinations (GRE)

Abstract

Most research on skill acquisition and retention focuses on the individual being tested. Yet sometimes another person is responsible for evaluating the individual’s performance. Here, we study the acquisition and retention of rater skill using data for the Graduate Record Examinations (GRE). Our work is based on the idea that response scoring, like other cognitive skills, will gradually improve with amount of practice, and decline with elapsed time since that practice occurred. These classic findings are the focus of a computational cognitive model called the Predictive Performance Equation (PPE). However, the generalizability of these findings to response scoring and the applicability of PPE to that domain have not yet been demonstrated. To address this issue, we leveraged a naturalistic dataset containing rating performance from over 23,000 sessions. Our analyses provide empirical support for PPE and establish a basis for using a model like PPE to personalize rater training requirements.