Effects of chatbot-assisted in-class debates on students’ argumentation skills and task motivation

Abstract

Recent advancements in artificial intelligence have led to the development of chatbots capable of engaging in argumentative dialogues and debates with human users. Although some studies have investigated the use of such chatbots to facilitate argumentation learning outside of the classroom, their integration into in-class learning activities remains largely unexplored. In this study, we developed a novel task design, chatbot-assisted in-class debates (CaIcD), for argumentation learning. In the task design, the students interacted with an argumentative chatbot named Argumate before engaging in debates with their classmates. During their interaction, the chatbot helped the students to generate ideas for supporting their position and predict opposing viewpoints. This study investigated the effects of CaIcD on students' argumentation skills and task motivation. Forty-four Chinese undergraduate students from two classes participated in this study. To examine the effects on argumentation skills in terms of structural complexity and argument quality, we used a pretest–posttest quasi-experimental design. Quade's test results revealed that participation in CaIcD enabled the students to use more claims, data, and warrants to generate arguments and participation in CaIcD led to more organised, sufficient, and elaborated arguments. However, no significant effects on overall structural complexity were found. Moreover, to understand the students' task motivation towards CaIcD, a within-subjects comparison design was employed. The results of the Wilcoxon signed-rank test indicated that the students had a higher level of enjoyment and exerted more effort when engaging in CaIcD than when performing conventional learning tasks. Moreover, the students perceived their performance in CaIcD to be as successful as that in conventional learning tasks, although the CaIcD task presented more challenges to them. However, no significant difference was observed in the students' perceived relevance of the two types of tasks to their argumentation learning. This study provides empirical evidence that integrating argumentative chatbots into classroom debates can lead to improved argumentation skills and higher task motivation among undergraduate students.