Where is the Learning in Learning Analytics? A Systematic Literature Review on the Operationalization of Learning-Related Constructs in the Evaluation of Learning Analytics Interventions

Justian Knobbout,Esther Van Der Stappen

doi:10.1109/tlt.2020.2999970

Justian Knobbout, Esther Van Der Stappen

Open Access

https://doi.org/10.1109/tlt.2020.2999970

Copy DOI

Abstract

Learning technologies enable interventions in the learning process aiming to improve learning. Learning analytics provides such interventions based on analysis of learner data, which are believed to have beneficial effects on both learning and the learning environment. Literature reporting on the effects of learning analytics interventions on learning allows us to assess in what way learning analytics improves learning. No standard set of operational definitions for learning affected by learning analytics interventions is available. We performed a systematic literature review of 1932 search hits, which yielded 62 key studies. We analyzed how affected learning was operationalized in these key studies and classified operational definitions into three categories: 1) learning environment ; 2) learning process; and 3) learning outcome . A deepening analysis yielded a refined classification scheme with 11 subcategories. Most of the analyzed studies relate to either learning outcome or learning process . Only nine of the key studies relate to more than one category. Given the complex nature of applying learning analytics interventions in practice, measuring the effects on a wider spectrum of aspects can give more insight into the workings of learning analytics interventions on the different actors, processes, and outcomes involved. Based on the results of our review, we recommend making deliberate decisions on the (multiple) aspects of learning one tries to improve by applying learning analytics. Our refined classification with examples of operational definitions may help both academics and practitioners doing so, as it allows for a more structured, grounded, and comparable positioning of learning analytics benefits.

Highlights

L EARNING technologies enable interventions in the learning process aiming to improve learning
The aim of this article was to provide an answer to the research question: In what way does existing literature on learning analytics interventions operationalize affected learning? The first conclusion is that, from 1932 search hits on learning analytics, only 62 describe quantitative, measurable effects of complete learning analytics cycles in authentic learning context
This is a noticeable shortcoming and in line with previous research that concluded that not enough studies make a connection to the stage of the learning analytics cycle, i.e., “not enough published work is making clear how the move will be made from researching the data to optimizing the learning” [5]

Summary

Introduction

L EARNING technologies enable interventions in the learning process aiming to improve learning. The Learning Analytics Cycle [2] describes the process of turning data into action and involves four steps: 1) learners generate data; 2) the infrastructure captures, collects, and stores this data; 3) the collected data are analyzed and visualized; and 4) feeding back these analytics and/or visualizations to stakeholders, such as learners and teachers. Such a learning analytics intervention is needed in order for learning analytics to have effect on learners. Learning analytics interventions can be defined as “the surrounding frame of activity through which analytic tools, data, and reports are taken up and used” [11]

Objectives

Methods

Results

Conclusion