Comparing the effectiveness of different methods to reduce the effect size of the practice effect in traffic psychological assessment

Abstract

Several theoretical models have been advanced in the literature to account for practice effects due to retesting. These models differ in their predictions regarding the extent to which item parameters can be assumed to be invariant across test administration sessions. Furthermore, the models also differ in their predictions regarding the effect of using computerized adaptive testing on the size of practice effects. The present study was conducted to test these predictions. A total of N = 891 test-takers were randomly assigned to one of six experimental conditions and solved different versions of a tachistoscopic perceptual speed tests at two time-points of measurement separated by approximately three months. The experimental conditions systematically varied regarding the use of identical or alternate retest forms and with regard to the use of either computerized adaptive test forms, or fixed-item linear test forms at the initial and/or retest session. Item response theory analyses indicated that practice effects can be explained in terms of an increase in test-specific abilities due to learning during test-taking. This conclusion was further corroborated by a comparison of practice effects across the six experimental conditions. In line with a model attributing practice effects to test-specific abilities due to learning during test-taking, practice effects were smallest in conditions in which computerized adaptive tests were administered at the initial test administration session. The pattern of practice effect sizes also contradicted predictions made on the basis of alternate theoretical models. Furthermore, computerized adaptive testing was more effective at reducing practice effects than alternate retest forms.