Automatic Enemy Identification—Are We There Yet?

Abstract

Enemy items refer to any two items that should not appear on the same test form (Weir, 2019). Accurately identifying enemy pairs is critical for ensuring the quality and fairness of exams, but it can also be challenging and time-consuming given the large number of possible item pairs in the exam item bank. Various enemy identification approaches have been explored to automate or semi-automate this task. In this process, the critical component is the encoding technique. The better the encoding technique captures the meaning of the sentences, the more accurate the similarity index and enemy classification results will be. This study focuses on evaluating the performance of a transformer-based model against the results from a string-based vector-space model (VSM) encoding technique under different research conditions for multiple-choice and multiple-response items used in a foundational IT certification exam. The results suggest that when using sufficient representative training data and conducting fine-tuning, the transformer-based model significantly outperforms the VSM model for enemy identification.