An investigation of mother tongue differential item functioning in a high-stakes computerized academic reading test

Abstract

A fundamental requirement of language assessments which is underresearched in computerized assessments is impartiality (fairness) or equal treatment of test takers regardless of background. The present study aimed to evaluate fairness in the Pearson Test of English (PTE) Academic Reading test, which is a computerized reading assessment, by investigating differential item functioning (DIF) across Indo-European (IE) and Non-Indo-European (NIE) language families. Previous research has shown that similarities between readers’ mother tongue and the second language being learned can advantage some test takers. To test this hypothesis, we analyzed data from 783 international test takers who took the PTE Academic test, using the partial credit model in Rasch measurement. We examined two main types of DIF: uniform DIF (UDIF), which occurs when an item consistently gives a particular group of test takers an advantage across all levels of ability, and non-uniform DIF (NUDIF), which occurs when the performance of test takers varies across the ability continuum. The results showed no statistically significant UDIF (p > 0.05), but identified 3 NUDIF items out of 10 items across the language families. A mother tongue advantage was not observed. Similarity in test takers’ level of computer and Internet skills, test preparation, and language policies could contribute to the finding of no UDIF. Post-hoc content analysis of items suggested that the decrease of mother tongue advantage for IE groups in high-proficiency groups and lucky guesses of low-ability groups may have contributed to the emergence of NUDIF items. Lastly, recommendations for investigating social and contextual factors are proposed.