The development and implementation of a computer adaptive progress test across European countries

Abstract

Longitudinal progress testing promotes self-directed deep learning across a full spectrum of knowledge, enabling early detection of underperformance and opportunities for remediation. Computer adaptive testing (CAT), where the difficulty of a test dynamically adjusts according to a test taker's ability, has benefits in a progress testing context, but significant resource and experience is required to develop appropriate test materials. This study describes how a transnational consortium from eight medical schools in five countries across Europe was formed to develop a computer adaptive progress test applicable across international curricula. 1,212 students from more than 40 nationalities took part in the study, of whom more than 70% were not native English speakers, though nearly all reported competence in English. A content map for an international assessment blueprint was agreed and a substantial bank of 1,127 English language progress test items was successfully calibrated after pilot testing to form the computer adaptive progress test (CA-PT) item bank. Results from the CA-PT pilot showed reliable convergence to stable estimates of ability, low standard errors of measurement and high test reliability for all participants. This study shows that an international collaborative consortium approach enables effective development of progress testing resources appropriate for computer adaptive testing, with potential for application across international borders and in populations where English is not the native language. Pooling resources internationally facilitates the comparison and development of appropriate assessment blueprints and the efficient generation of high-quality assessment items. • Computer adaptive testing is applicable for progress testing locally and internationally. • Consortium approaches improve productivity in test blueprint and item development. • Collaboration can generate a large calibrated item bank efficiently with little redundancy. • Computer adaptive progress testing improves individual test reliability. • Adaptive algorithms sampling items from blueprints enhances personalised assessment for learning.