Can ChatGPT effectively complement teacher assessment of undergraduate students’ academic writing?

Abstract

The integration of ChatGPT as a supplementary tool for writing instruction has gained traction. However, uncertainties persist regarding how ChatGPT complements teacher assessment and the overall effectiveness of this combined approach. To address this, we conducted a mixed-methods investigation involving 46 undergraduate students from a research university in southern China, engaging them in a Chinese academic writing task. The intraclass correlation coefficient results revealed ChatGPT’s efficiency in scoring students’ writing, showing moderate to good consistency with teacher evaluations. A paired sample t-test unveiled significant differences in feedback quantity and types between ChatGPT and teacher assessments. Drawing from both interview data and quantitative findings, the study uncovers three ways in which ChatGPT complements teacher assessment, benefiting students with various writing proficiency levels: (1) fostering deeper comprehension of teacher assessments among students, (2) encouraging students to make judgments regarding feedback, and (3) promoting independent thinking about revisions. This study contributes to a more comprehensive understanding of the role of ChatGPT within the context of a combined assessment approach. It underscores that certain inherent weaknesses in ChatGPT’s functioning can paradoxically lead to favorable outcomes. By shedding light on the synergy between ChatGPT and teacher assessments, this research seeks to inform and enhance writing instruction in higher education.