Designing a Human-centred Learning Analytics Dashboard In-use

Abstract

Despite growing interest in applying human-centred design methods to create learning analytics (LA) systems, most efforts have concentrated on initial design phases, with limited exploration of how LA tools and practices can co-evolve during the actual learning and teaching activities. This paper examines how a human-centred LA dashboard can be further refined and adapted by teachers while actively using it in a real-world scenario (i.e., design-in-use), beyond its intended design (i.e., design-for-use). We use instrumental genesis as a theoretical lens to analyze the temporary and permanent instrumentalization of design features and individual and collective instrumentation of the LA dashboard. The analysis of semi-structured individual interviews with five nursing teachers who used an LA dashboard to guide team reflections with 224 students (56 teams) revealed technical and pedagogical changes that occurred in both the system’s features (instrumentalization) and teaching practices (instrumentation). We found that teachers adopted the LA dashboard beyond initially intended ways by (i) providing emotional support with the analytics, (ii) reducing details in AI-powered data visualizations for easier comprehension, (iii) creating data narratives to address data limitations, and (iv) collectively developing new practices to use the LA dashboard for co-teaching. Therefore, teachers’ design-in-use of the LA dashboard highlights the ongoing need for design improvements to address challenges posed by dynamic data and complex algorithms underlying AI and analytics interfaces.