Gamification suffers from the novelty effect but benefits from the familiarization effect: Findings from a longitudinal study

Luiz Rodrigues,Armando M Toda,Leandro Silva Galvão Carvalho,Filipe D Pereira,Elaine H T Oliveira,David Fernandes,Marcela Pessoa,Alexandra I Cristea,Seiji Isotani,Paula T Palomino

doi:10.1186/s41239-021-00314-6

Abstract

There are many claims that gamification (i.e., using game elements outside games) impact decreases over time (i.e., the novelty effect). Most studies analyzing this effect focused on extrinsic game elements, while fictional and collaborative competition have been recently recommended. Additionally, to the best of our knowledge, no long-term research has been carried out with STEM learners from introductory programming courses (CS1), a context that demands encouraging practice and mitigating motivation throughout the semester. Therefore, the main goal of this work is to better understand how the impact of a gamification design, featuring fictional and competitive-collaborative elements, changes over a 14-week period of time, when applied to CS1 courses taken by STEM students (N = 756). In an ecological setting, we followed a 2x7 quasi-experimental design, where Brazilian STEM students completed assignments in either a gamified or non-gamified version of the same system, which provided the measures (number of attempts, usage time, and system access) to assess user behavior at seven points in time. Results indicate changes in gamification’s impact that appear to follow a U-shaped pattern. Supporting the novelty effect, the gamification’s effect started to decrease after four weeks, decrease that lasted between two to six weeks. Interestingly, the gamification’s impact shifted to an uptrend between six and 10 weeks after the start of the intervention, partially recovering its contribution naturally. Thus, we found empirical evidence supporting that gamification likely suffers from the novelty effect, but also benefits from the familiarization effect, which contributes to an overall positive impact on students. These findings may provide some guidelines to inform practitioners about how long the initial contributions of gamification last, and how long they take to recover after some reduction in benefits. It can also help researchers to realize when to apply/evaluate interventions that use gamification by taking into consideration the novelty effect and, thereby, better understand the real impact of gamification on students’ behavior in the long run.

Highlights

Game elements have been widely added outside games, approach known as gamification, with the goal of affecting user behavior Deterding et al (2011); Helmefalk (2019)
After the end of the decline, our results show an increase in gamification impact for all measures, which either continued to increase or achieved a rough plateau (IDE usage and system access)
Gamification is an educational innovation that has been widely used in several domains

Summary

Introduction

Game elements have been widely added outside games, approach known as gamification, with the goal of affecting user behavior Deterding et al (2011); Helmefalk (2019). Studies have provided empirical evidence supporting the novelty effect in gamified learning Those studies show that, for instance, users’ perceived benefits from a gamified service decreased as the time using that service increased Koivisto and Hamari (2014) and that there are cases when gamification’s impact on student behavior and motivation changed after an initial positive effect Sanchez et al (2020); Rodrigues et al (2021). Such findings question whether gamification can lead to long-lasting benefits. Researchers have called for longitudinal studies tracking gamification’s impact over long timeperiods Bai et al (2020); Alsawaier (2018); Kyewski and Krämer (2018)

Objectives

Methods

Results

Discussion

Conclusion