Comparing countdown- and IRT-based approaches to computerized adaptive personality testing.

Abstract

Computerized adaptive testing (CAT) is an emerging technology in the personality assessment literature given the greater efficiency it affords compared with traditional methods. However, few studies have directly compared the efficiency and validity of 2 competing methods for personality CAT: (a) methods based on item response theory (IRT-CAT) versus (b) methods based on the countdown method (CM-CAT). To that end, we conducted real-data simulations with previously collected responses (N = 8,690) to the Schedule for Nonadaptive and Adaptive Personality (SNAP). Three CAT algorithms (IRT-CAT, IRT-CAT with 5-item minimum, CM-CAT) were evaluated for item savings, classification accuracy, and convergent/discriminant validity. All CATs yielded lower classification accuracy and validity than traditional testing but required 18%-86% fewer items. Ultimately, the IRT-CAT, with minimum 5-item requirement, struck the most ideal balance of highest item savings, and generally fewer costs to validity and accuracy. These results confirm findings regarding item savings trends from previous CAT studies. In addition, this study provides a model for how the validity and precision of CATs may be compared across personality assessments.