Do individual differences in test preparation compromise the measurement fairness of admission tests?

Abstract

The existence of individual differences in the use of test preparation has raised concerns regarding the fair and valid use of admission tests. Measurement specialists argued that the extent to which individual differences in test preparation affects the measurement fairness and construct validity of admission tests depends on the processes that lead to an increase in admission tests score due to test preparation. Four theoretical models have been advanced in the literature to account for the effect of test preparation on admission test scores. The four theoretical models make competing predictions with regard to the processes that lead to an increase in admission test scores and the extent to which measurement fairness can be assumed across test-takers differing in test preparation at the time-point of the actual admission exam. In the present article, latent class analysis was used to model individual differences in the use of test preparation methods used by applicants to a medical university (N=1768). Four latent classes of test-takers were identified that differed qualitatively and quantitatively in terms of test preparation. Item response theory analyses and multi-group means and covariance structure analyses indicated strict measurement invariance across the four latent classes at the level of the individual subtests. However, group differences in test-takers admission test scores were not related to differences in either psychometric g, or general natural science knowledge (Gk-ns), respectively. The findings were consistent with theoretical models that attribute the effect of test preparation on test-takers' admission test scores to an increase in test-specific cognitive abilities and/or domain-specific knowledge.