Evaluating user experience of adaptive digital educational games with Activity Theory

Abstract

Adaptive digital educational games (DEGs) providing players with relevant interventions can enhance gameplay experience. This advance in game design, however, renders the user experience (UX) evaluation of DEGs even more challenging. To tackle this challenge, we developed a four-dimension evaluation framework (i.e., gaming experience, learning experience, adaptivity, and usability) and applied it to an empirical study with a DEG on teaching geography. Mixed-method approaches were adopted to collect data with 16 boys aged 10–11. Specifically, a so-called Dyadic User Experience Tests (DUxT) was employed; participants were paired up to assume different roles during gameplay. Learning efficacy was evaluated with a pre-post intervention measurement using a domain-specific questionnaire. Learning experience, gaming experiences and usability were evaluated with intensive in situ observations and interviews guided by a multidimensional scheme; content analysis of these transcribed audio data was supplemented by video analysis. Effectiveness of adaptivity algorithms was planned to be evaluated with automatic logfiles, which, unfortunately, could not be realised due to some technical problem. Nonetheless, the user-based data could offer some insights into this issue. Furthermore, we attempted to bridge the existing gap in UX research – the lack of theoretical frameworks in understanding user experience – by adopting Engeström's (1987) extended framework of Activity Theory (AT) that provides contextual information essential for understanding contradictions and breakdowns observed in the interactions between the game players. The dyadic gameplay setting allows us to explore the issue of group UX. Implications for further applications of the AT framework in the UX research, especially the interplay between evaluation and redesign (i.e., downstream utility of UX evaluation methods), are discussed.