Utilizing artificial intelligence to support analyzing self-regulated learning: A preliminary mixed-methods evaluation from a human-centered perspective

Abstract

Analyzing the self-regulatory process of complex science learning is a serious challenge as it takes considerable time to train coders and do real-time assessment of a learner's verbatim transcript. Thus, the aim of this study was to investigate the potential and opportunities of artificial intelligence (AI) methods to analyze self-reporting protocols for recognizing cognitive and metacognitive strategies of self-regulated learning. Sixty-six participants were recruited to evaluate the quality of given scientific explanations while self-reporting their interpreting and reasoning processes. The self-reported protocols were further coded and categorized as cognitive or metacognitive activities for training and evaluating an AI model. Long Short-Term Memory, an AI classifier, was employed to predict the rich narrative texts expressed by learners on using strategies in complex science tasks. Quantitative analysis was conducted to evaluate the performance of the classifier. Results suggested promising accuracy/consistency between human-based and the AI classifier. In addition, two design factors, AI structure and dropout rate, did not significantly impact the performance. Qualitative examinations of discrepancies between human and AI classifier revealed that length of segments and segments including a phrase or words with temporal cues could potentially influence the accuracy of AI judgments. Overall, The AI classifier yielded a fair performance demonstrating acceptable accuracy in the prediction of cognitive or metacognitive strategies with a limited dataset for a total of merely 104 protocols from 66 participants. Our qualitative observations that attempt to explain sources of human-computer discrepancies may shed light on future improvement for AI-based methods. Implications of AI for self-regulated digital learning are discussed.