Predicting the Item Difficulty of a Simulated CSAT English Test Based on Corpus Analysis

Abstract

This study investigates the relationship between linguistic features and item difficulty of a simulated College Scholastic Ability Test (CSAT) English test, based on a corpus analysis. The test used for the present study was a simulated CSAT English test administered in June 2020. Item difficulty data was collected from 101,386 students who took the test. For the corpus analysis, lexical and syntactic variables were measured by the Lexical Complexity Analyzer (LCA) and the L2 Syntactic Complexity Analyzer (L2SCA), the computational tool and were correlated with item difficulty (dependent variable) for 41 test items. According to the correlation analysis, one lexical variable and all syntactic variables were found to be significantly correlated with item difficulty. Also, the results of the multiple regression indicate that lexical sophistication and particular structures are related to item difficulty, explaining for approximately 55.1% of the variance in item difficulty. The results showed that the variables identified in the current study were explanatory in terms of predicting item difficulty of the CSAT English test. Therefore, the findings of this study have pedagogical implications for test developers and education policy makers in Korea, with regard to text characteristics and test difficulty.