Theory-driven game-based assessment of general cognitive ability: Design theory, measurement, prediction of performance, and test fairness.

Abstract

Games, which can be defined as an externally structured, goal-directed type of play, are increasingly being used in high-stakes testing contexts to measure targeted constructs for use in the selection and promotion of employees. Despite this increasing popularity, little is known about how theory-driven game-based assessments (GBA), those designed to reflect a targeted construct, should be designed, or their potential for achieving their simultaneous goals of positive reactions and high-quality psychometric measurement. In the present research, we develop a theory of GBA design by integrating game design and development theory from human-computer interaction with psychometric theory. Next, we test measurement characteristics, prediction of performance, fairness, and reactions of a GBA designed according to this theory to measure latent general intelligence (g). Using an academic sample with GPA data (N = 633), we demonstrate convergence between latent GBA performance and g (β = .97). Adding an organizational sample with supervisory ratings of job performance (N = 49), we show GBA prediction of both GPA (r = .16) and supervisory ratings (r = .29). We also show incremental prediction of GPA using unit-weighted composites of the g test battery beyond that of the g-GBA battery but not the reverse. We also show the presence of similar adverse impact for both the traditional test battery and GBA but the absence of differential prediction of criteria. Reactions were more positive across all measures for the g-GBA compared to the traditional test battery. Overall, results support GBA design theory as a promising foundation from which to build high-quality theory-driven GBAs. (PsycInfo Database Record (c) 2022 APA, all rights reserved).