Robustness of Computer Adaptive Tests to the Presence of Item Preknowledge: A Simulation Study

Abstract

Item preknowledge describes a scenario where candidates may have access to the some of the test items prior to the test administration. This involves the sharing of test materials and/or answers and it is difficult to identify the individuals with item preknowledge or the shared materials of test. Nevertheless, it is essential to investigate the ‘item preknowledge’ problem because it can significantly affect the validity of test results. It is believed that traditional linear tests are more robust to this type of aberrant response behavior than adaptive tests. In this context, the aim of this study is to examine the effect of item preknowledge on computer adaptive tests and to try to identify the conditions under which adaptive tests are most resistant to the item preknowledge. With this purpose, a Monte Carlo simulation study was performed and 28 different conditions were examined. The results of the study indicated the EAP estimation method provided better measurement precision than ML over all conditions. When 2PL and 3PL IRT models were compared, it was observed that 2PL had higher precision at most of the conditions. However, when the aberrancy ratio increased and reached to the 20% for both individuals and items, 3PL outperformed the 2PL model and gave the best results with the EAP combination. The results were discussed in line with literature on item preknowledge and CAT and implications for practitioners and further research were provided.