The effect of providing learning analytics on student behaviour and performance in programming: a randomised controlled experiment

Jan Hellings,Carla Haelermans

doi:10.1007/s10734-020-00560-z

Abstract

We use a randomised experiment to study the effect of offering half of 556 freshman students a learning analytics dashboard and a weekly email with a link to their dashboard, on student behaviour in the online environment and final exam performance. The dashboard shows their online progress in the learning management systems, their predicted chance of passing, their predicted grade and their online intermediate performance compared with the total cohort. The email with dashboard access, as well as dashboard use, has positive effects on student behaviour in the online environment, but no effects are found on student performance in the final exam of the programming course. However, we do find differential effects by specialisation and student characteristics.

Highlights

Many policymakers and higher education institutes (HEIs) are concerned with the low completion rates in higher education, which are often ascribed to the high number of student dropouts in the first years of the bachelor education
Given the increasing use of online environments, in combination with the desire to increase self-regulation and involvement of students, one option is to use visualisation technique such as dashboards based on learning analytics, in combination with early warning systems, to signal to students how they are doing on their activities in online environments and to remind them to study regularly (Bos 2016 ) (Lauría et al 2013), and by sending regular emails with access links to the dashboard and the learning analytics, which might serve as a reminder for their coursework
We find that the weekly email and corresponding dashboard access, as well as dashboard use, have positive effects on student performance in the online environment during the course, but not on student behaviour in the online environment, or on student performance in the final exam or their chances of passing the programming course

Summary

Introduction

Many policymakers and higher education institutes (HEIs) are concerned with the low completion rates in higher education, which are often ascribed to the high number of student dropouts in the first years of the bachelor education. It is often said that a lack of motivation and self-regulatory skills from the students might be the reason for the high dropout numbers (Kitsantas et al 2008), as well as that they have difficulties predicting or knowing during the course, whether they are doing well or not. These issues are prevalent in higher education in general, but in engineering education. Given the increasing use of online environments, in combination with the desire to increase self-regulation and involvement of students, one option is to use visualisation technique such as dashboards based on learning analytics, in combination with early warning systems, to signal to students how they are doing on their activities in online environments and to remind them to study regularly (Bos 2016 ) (Lauría et al 2013), and by sending regular emails with access links to the dashboard and the learning analytics, which might serve as a reminder for their coursework (similar to what has been done previously in the literature, see, e.g. Beattie et al 2018; Oreopoulos and Petronijevic 2018)

Objectives

Results

Conclusion