A Three-Step DIF Analysis of a Reading Comprehension Test Across Regional Dialects to Improve Test Score Validity

Abstract

ABSTRACT In sociolinguistic contexts where standardized languages coexist with regional dialects, the study of differential item functioning is a valuable tool for examining certain linguistic uses or varieties as threats to score validity. From an ecological perspective, this paper describes three stages in the study of differential item functioning (DIF): detection, comprehension and evaluation of its consequences. The research focuses on a reading comprehension test in Basque with 1,943 respondents representing two regional dialects. After assessing the dimensionality of the data fitting unidimensional and bifactor models, we start with a multi-method approach (Item Response Theory, Mantel-Haenszel, Standardization) to detect DIF. In a second step, we study the causes of DIF based on a qualitative methodology approach built on the focus group technique. Finally, the impact of DIF on the test level is evaluated through indicators of differential test functioning (DTF) derived from the IRT, and comparisons between situations that consider the presence/absence of DIF. The paper stresses the importance of the comprehensive study of DIF in language accreditation tests as a tool for improving score validity.