Does Scientific Evaluation Matter? Improving Digital Simulation Games by Design-Based Research

Monika Oberle,Sven Ivens

doi:10.3390/socsci9090155

Abstract

Grounded in a design-based research approach, the aim of this article is to determine whether scientific evaluations help to (a) identify and fix problems in educational interventions and (b) eventually foster a more effective and positive evaluated intervention. Therefore, data from a longer-term evaluation of short digital simulation games about the European Parliament for civic education in schools were used. The data included three cycles of interventions with pre- and post-evaluations starting with the first prototype in 2015/2016 (n = 209), the second cycle in 2017/18 (n = 97), and the last one in 2019/20 (n = 222). After each evaluation, major problems and critiques regarding the simulation game were discussed with the developers, and changes were implemented in the game design. The four most important problems, the processes by which they were improved and the reactions of the participants in the following evaluations are pointed out in the article. A comparison of the last and first evaluation cycle showed an overall improvement of the simulation game regarding its effectiveness in transferring EU knowledge and the participants’ general satisfaction with the simulation game. This study underlines the value of the design-based research approach for developing educational interventions and can be useful for further work on civic education measures and the implementation of digital simulation games.

Highlights

The effectiveness of active citizenship education programs, tools, and interventions are often measured and evaluated by scientists in many different ways, reaching from qualitative interviews or self-reflections to experimental studies and large n survey studies
What happens after the evaluation is done and the article is published? Is measuring and evaluating civic education programs and teaching tools an end in itself, or does doing so help to increase the effectiveness of these programs? This article investigates the usefulness of scientific evaluation in civic education according to the design-based research approach
This quite new research approach goes a step further, as it focuses on evaluating educational interventions and on enhancing them according to the collected evaluation data and evaluating them again to see if the changes were effective until the educational intervention leads to a satisfactory result (Anderson and Shattuck 2012)

Summary

Introduction

The effectiveness of active citizenship education programs, tools, and interventions are often measured and evaluated by scientists in many different ways, reaching from qualitative interviews or self-reflections to (quasi-) experimental studies and large n survey studies. This article investigates the usefulness of scientific evaluation in civic education according to the design-based research approach. This quite new research approach goes a step further, as it focuses on evaluating educational interventions and on enhancing them according to the collected evaluation data and evaluating them again to see if the changes were effective until the educational intervention leads to a satisfactory result (Anderson and Shattuck 2012). The design-based research approach presented here uses data from an ongoing evaluation study about short digital simulation games in civic education (Oberle et al 2017). It has recently been used for research on subject didactics in school (Peters and Roviró 2017), on higher education (Ford et al 2017), blended learning (Ustun and Tracey 2020), and simulation games (Koivisto et al 2018)

Results

Discussion

Conclusion