Quantitative differences in retest effects across different methods used to construct alternate test forms

Abstract

Allowing respondents to retake a cognitive ability test has shown to increase their test scores. Several theoretical models have been proposed to explain this effect, which make distinct assumptions regarding the measurement invariance of psychometric tests across test administration sessions with regard to narrower cognitive abilities and general mental ability. We modeled retest effects in four psychometric tests as a function of specific retest form and general mental ability in order to compare the validity of these models and their generalizability across three different kinds of retest forms. To do so automatic item generation was used to construct two kinds of alternate retest: (1) isomorphic retests and (2) psychometrically matched retests. A total of N=358 respondents completed all four measures twice, receiving either identical retest forms, isomorphic retest forms or psychometrically matched retest forms at the second test administration session. Item response theory modeling supported strict measurement invariance across all test forms and time-points of measurement but indicated variation in respondents' retest score gains due to individual differences in general mental ability and the kind of retest form used. In general, retest effects were more pronounced for high-g respondents, identical retests and isomorphic retest forms and for mental rotation and algebra word problems. Latent mean and covariance structure analyses indicated that retesting did not affect the g-factor saturation of the four cognitive ability tests but revealed that retest score gains were hollow with respect to psychometric g.