Group-level analysis of engagement poorly reflects individual students’ processes: Why we need idiographic learning analytics

Abstract

A central assumption of the scientific method is that inferences derived from group-level analysis align with and generalize to the individual level. This study was conducted to put this assumption to the test to examine if and to what extent our analysis, inferences, and assumptions hold, and which variables generalize from the group to the individual level. We use engagement as the underpinning of this study. However, the same methods and questions apply elsewhere. The study included 238 students over six courses and applied the latest advances in psychological networks. Two networks were estimated using the same data: a between-person model that captures the group-level engagement and a within-person model that captures the within-person processes. The results showed that there were significant differences between the two networks and a lack of generalizability regarding regularity, academic achievement, and online disengagement. Such findings cast doubts on inferences drawn from group-level data about our understanding of learners’ performance or engagement, or to design personalized interventions. More attention and efforts are needed to further model within-person processes to understand, and possibly deliver precise personalized support and interventions that are more generalizable and truly personalized.