Learning Analytics and the Measurement of Learning Engagement

Abstract

The measurement of learning engagement is a major research theme, both in the learning analytics community and the broader area of educational research. The complexity of conceptualizing as well as operationalizing the construct of engagement generates a wide range of instruments, such as self-report surveys, log data from technology-enhanced learning systems, think-aloud, and tests. In this empirical work, we investigate the alignment of behavioural traces of engagement with self-report measures and their impact on academic performance. The unique contribution of this study is the integration of temporal, behavioural, affective, and cognitive dimensions of engagement by combining digital traces at three different learning phases with self-report, formative as well as summative assessments. Using a two-step cluster analysis based on data from 1027 undergraduate students in a first-year 8-week statistics course, we identified four distinct temporal engagement patterns (i.e. nonactive, active before tutorial, active before quiz, and active before exams). Our analysis showed that early engagement (i.e. before tutorial) was significantly associated with course performance and self-report measures, while late engagement patterns had weaker correlations. This study shed further lights on a potential source of heterogeneity and collinearity in engagement measures (i.e. timing of engagement) that should be accounted for in learning analytics model. In order to design effective intervention, it is crucial to consider different profiles of learners based on their engagement patterns as well as the temporal relation between trace data, self-report, and academic performance.