Construction and validation of a game-based intelligence assessment in minecraft

Abstract

Video games are a promising tool for the psychometric assessment of cognitive abilities. They can present novel task types and answer formats, they can record process data, and they can be highly motivating for test takers. This paper introduces the first game-based intelligence assessment implemented in Minecraft, an exceptionally popular video game with more than 200 m copies sold. A matrix-based pattern completion task (PC), a mental rotation task (MR) and a spatial construction task (SC) were implemented in the three-dimensional, immersive environment of the game. PC was intended as a measure of inductive reasoning, whereas MR and SC were measures of spatial ability. We tested 129 children aged 10–12 years old on the Minecraft-based tests as well as equivalent pen-and-paper tests. All three scales fit the Rasch model and were moderately reliable. Factorial validity was good with regard to the distinction between PC and SC, but no distinct factor was found for MR. Convergent validity was good as abilities measured with Minecraft and conventional tests were highly correlated at the latent level (r = 0.72). Subtest-level correlations were in the moderate range. Furthermore, we found that behavioral log-data collected from the game environment was highly predictive of performance in the Minecraft test and, to a lesser extent, also predicted scores in conventional tests. We identify a number of behavioral features associated with spatial reasoning ability, demonstrating the utility of analyzing granular behavioral data in addition to traditional response formats. Overall, our findings indicate that Minecraft is a suitable platform for game-based intelligence assessment and encourage future work aiming to explore game-based problem solving tasks that would not be feasible on paper or in conventional computer-based tests.