Development and Application of Detection Indices for Measuring Guessing Behaviors and Test-Taking Effort in Computerized Adaptive Testing

Abstract

This study examined the amount of time that different ability-level examinees spend on questions they answer correctly or incorrectly across different pretest item blocks presented on a fixed-length, time-restricted computerized adaptive testing (CAT). Results indicate that different ability-level examinees require different amounts of time to answer pretest items, that is, field-test items, embedded in different item locations. Also, guessing behaviors and test-taking effort (TTE), measured using detection indices developed from this study, varied across the pretest items embedded in different locations. With these indices, unique response time patterns (RTP) and guessing patterns (GP) were identified for six ability groups. These indices could also be used to monitor the amount of effort and the guessing behaviors of examinees taking a CAT. The degree of the detection consistency among these indices could be used as reliability evidence to evaluate how reliable the indices are in detecting unusual test-taking behaviors. Once reliable detection was confirmed, a warning sign prompted from the computer screen could either encourage the examinee to take appropriate TTE or stop guessing so that the examinees could receive more accurate estimates of their ability.