Artificial intelligence in history education. Linguistic content and complexity analyses of student writings in the CAHisT project (Computational assessment of historical thinking)

Abstract

The use of standardized test formats in the assessment of historical competencies has recently come under severe criticism, especially in the United States, where standardized tests are particularly common. History researchers have argued that open-ended items are more appropriate for assessment. However, providing largescale evaluations of open-ended answers is time consuming and poses challenges regarding the objectivity, validity, and replicability of ratings. To address this issue, we investigated the extent to which computer-based evaluation methods are suitable for evaluating student answers by combining qualitative methods from history education research with quantitative, computer-based linguistic analyses. In two studies, we analyzed data from an intervention study in which 962 students (ninth graders) completed seven open-ended tasks. In Study 1, we investigated the extent to which task complexity could be predicted from the linguistic complexity of the students' answers. In Study 2, we conducted an automatic content assessment by aligning student answers with predefined target answers and compared automatic scores with human ratings that were based on an elaborate evaluation scheme we developed. In our first study, we identified several linguistic features of students’ answers that successfully predicted task complexity. In our second study, the correlations between manual and computational task scores were encouraging. However, our human interrater agreement left room for improvement and demonstrated the challenges of reliably applying explicit evaluation criteria to open tasks. Overall, our findings illustrate that the combination of qualitative methods from history education research and quantitative computational-linguistic analyses may support the large-scale evaluation of open tasks.