Game-based Assessment Research Articles

Contemplating modes of assessing citizenship competences

Assessment of citizenship competences has become common practice following the statutory assignment of citizenship education to schools in many countries. Assessment can serve various goals. The suitability of various types of instruments depends on the alignment with the goals intended. In this paper we evaluate four types of instruments and their suitability to assess citizenship competences. Tests and questionnaires, portfolios, game-based assessment, and vignettes are each evaluated in terms of seven attributes relevant to instruments aiming to assess students’ citizenship competences. Our results indicate no single type of instrument aligns with all attributes, and expanding the range of available instruments appears the best way forward so that educators and researchers can make a choice that fits their purpose. The analysis presented provides further insight into the strengths and weaknesses of particular assessment types.

Validation of a Computerized, Game-based Assessment Strategy to Measure Training Effects on Motor-Cognitive Functions in People With Dementia.

BackgroundExergames often used for training purpose can also be applied to create assessments based on quantitative data derived from the game. A number of studies relate to these use functionalities developing specific assessment tasks by using the game software and provided good data on psychometric properties. However, (1) assessments often include tasks other than the original game task used for training and therefore relate to similar but not to identical or integrated performances trained, (2) people with diagnosed dementia have insufficiently been addressed in validation studies, and (3) studies did commonly not present validation data such as sensitivity to change, although this is a paramount objective for validation to evaluate responsiveness in intervention studies.ObjectiveSpecific assessment parameters have been developed using quantitative data directly derived from the data stream during the game task of a training device (Physiomat). The aim of this study was to present data on construct validity, test–retest reliability, sensitivity to change, and feasibility of this internal assessment approach, which allows the quantification of Physiomat training effects on motor-cognitive functions in 105 multimorbid patients with mild-to-moderate dementia (mean age 82.7±5.9).MethodsPhysiomat assessment includes various tasks at different complexity levels demanding balance and cognitive abilities. For construct validity, motor-cognitive Physiomat assessment tasks were compared with established motor and cognitive tests using Spearman’s rank correlations (rs). For test–retest reliability, we used intra-class correlations (ICC3,1) and focused on all Physiomat tasks. Sensitivity to change of trained Physiomat tasks was tested using Wilcoxon statistic and standardized response means (SRMs). Completion rate and time were calculated for feasibility.ResultsAnalyses have mostly shown moderate-to-high correlations between established motor as well as cognitive tests and simple (rs=−.22 to .68, P ≤.001-.03), moderate (rs=−.33 to .71, P ≤.001-.004), and complex motor-cognitive Physiomat tasks (rs=−.22 to .83, P ≤.001-.30) indicating a good construct validity. Moderate-to-high correlations between test and retest assessments were found for simple, moderate, and complex motor-cognitive tasks (ICC=.47-.83, P ≤.001) indicating good test–retest reliability. Sensitivity to change was good to excellent for Physiomat assessment as it reproduced significant improvements (P ≤.001) with mostly moderate-to-large effect sizes (SRM=0.5-2.0) regarding all trained tasks. Completion time averaged 25.8 minutes. Completion rate was high for initial Physiomat measures. No adverse events occurred during assessment.ConclusionsOverall, Physiomat proved to have good psychometric qualities in people with mild-to-moderate dementia representing a reliable, valid, responsive, and feasible assessment strategy for multimorbid older adults with or without cognitive impairment, which relates to identical and integrated performances trained by using the game.

Content validity of game-based assessment: case study of a serious game for ICT managers in training

Serious games foster the acquisition of complex problem-solving skills. Assessment of such skills should be in line with instruction, and within a serious game environment its content validity should equal face-to-face assessment. Research on assessment in serious gaming has remained rather scarce. This article shows how assessment can be implemented in serious gaming in a way that assures content validity. The core of the authors’ validation method entails mapping learning activities (as contained in the game scenario) on performance indicators and outputs (as derived from formal attainment levels). They present how they have elaborated and applied the method for an assessment game for ICT managers in secondary vocational education. They describe the procedure and extent to which this assessment is content-valid compared to face-to-face assessment.

A Serious Game for Clinical Assessment of Cognitive Status: Validation Study.

BackgroundWe propose the use of serious games to screen for abnormal cognitive status in situations where it may be too costly or impractical to use standard cognitive assessments (eg, emergency departments). If validated, serious games in health care could enable broader availability of efficient and engaging cognitive screening.ObjectiveThe objective of this work is to demonstrate the feasibility of a game-based cognitive assessment delivered on tablet technology to a clinical sample and to conduct preliminary validation against standard mental status tools commonly used in elderly populations.MethodsWe carried out a feasibility study in a hospital emergency department to evaluate the use of a serious game by elderly adults (N=146; age: mean 80.59, SD 6.00, range 70-94 years). We correlated game performance against a number of standard assessments, including the Mini-Mental State Examination (MMSE), Montreal Cognitive Assessment (MoCA), and the Confusion Assessment Method (CAM).ResultsAfter a series of modifications, the game could be used by a wide range of elderly patients in the emergency department demonstrating its feasibility for use with these users. Of 146 patients, 141 (96.6%) consented to participate and played our serious game. Refusals to play the game were typically due to concerns of family members rather than unwillingness of the patient to play the game. Performance on the serious game correlated significantly with the MoCA (r=–.339, P <.001) and MMSE (r=–.558, P <.001), and correlated (point-biserial correlation) with the CAM (r=.565, P <.001) and with other cognitive assessments.ConclusionsThis research demonstrates the feasibility of using serious games in a clinical setting. Further research is required to demonstrate the validity and reliability of game-based assessments for clinical decision making.

Measuring problem solving skills via stealth assessment in an engaging video game

We used stealth assessment, embedded in a game called Use Your Brainz (a slightly modified version of Plants vs. Zombies 2), to measure middle-school students’ problem solving skills. We began by developing a problem solving competency model based on a review of the relevant literature. We then identified in-game indicators that would provide evidence about students’ levels on various problem-solving facets. Our problem solving model was then implemented in the game via Bayesian networks. To validate the stealth assessment, we collected data from students who played the game-based assessment for three hours and completed two external problem solving measures (i.e., Raven’s Progressive Matrices and MicroDYN). Results indicated that the problem solving estimates derived from the game significantly correlated with the external measures, which suggests that our stealth assessment is valid. Our next steps include running a larger validation study and developing tools to help educators interpret the results of the assessment, which will subsequently support the development of problem solving skills.

A pilot study: a computer game-based assessment of visual perspective taking of four children with autism with high support needs

High support need and minimally verbal individuals with autism have received less attention in research in comparison to so-called higher functioning individuals with autism. As computers motivate individuals with autism, a game with a positive user experience was altered into a level 1 perspective-taking task in which advancement was contingent on eye contact. A case-controls design was used to see whether participants had impaired perspective taking and would they benefit from additional cues. Four high support need and minimally verbal children with autism played the game in their school environment. It was found that only one child with autism made more errors than controls using the eye cues. No child with autism benefitted from the additional cues, whereas the control group did. The results suggest that positive contexts may reveal more about individuals and their skills, and that individual level analysis can provide insights about autism and about the individuals with autism.

Game-based assessment: Face validity, fairness perception, and impact on employer’s brand image

Game-based assessment: Face validity, fairness perception, and impact on employer’s brand image

Applying Evidence-Centered Design for the Development of Game-Based Assessments in Physics Playground

Game-based assessment (GBA) is a specific use of educational games that employs game activities to elicit evidence for educationally valuable skills and knowledge. While this approach can provide individualized and diagnostic information about students, the design and development of assessment mechanics for a GBA is a nontrivial task. In this article, we describe the 10-step procedure that the design team of Physics Playground (formerly known as Newton's Playground) has established by adapting evidence-centered design to address unique challenges of GBA. The scaling method used for Physics Playground was Bayesian networks; thus this article describes specific actions taken for the iterative process of constructing and revising Bayesian networks in the context of the game Physics Playground.

Using video games to combine learning and assessment in mathematics education

One problem with most education systems is that learning and (summative) assessment are generally treated as quite separate things in schools. We argue that video games can provide an opportunity to combine these processes in an engaging and effective way. The present study focuses on investigating the effectiveness and the assessment power of two different mathematics video games, Semideus and Wuzzit Trouble. In the current study, we validated the Semideus game as a rational number test instrument. We used it as a pre- and a post-test for a three-hour intervention in which we studied the effectiveness of Wuzzit Trouble, a game built on whole number arithmetic and designed to enhance mathematical thinking and problem solving skills. The results showed that (1) games can be used to assess mathematical knowledge validly, and (2) even short game-based interventions can be very effective. Based on the results, we argue that game-based assessment can create a more complete picture of mathematical knowledge than simply measuring students' accuracy, providing indicators of student misconceptions and conceptual change processes

Employing game analytics techniques in the psychometric measurement of game-based assessments with dynamic content

The adaptation Game-Based Assessment (GBA) (Mislevy et al., 2014) has been growing in the last years backed by video games’ capability of offering a task model to assess learners’ complex knowledge. Since the variables generated from such performances are not directly interpretable, assessment frameworks such as Evidence-Centred Design (ECD) came into play (Mislevy & Almond, 2003). In this work we show our initial findings when using game analytics techniques (Saif El-Nasr et al., 2013) such as play metrics to analyse players’ performance in an open world 3D game for traffic education. The results show that play metrics can be used in cases where game has a dynamic user-generated content of unknown structure. Additionally, we discuss how these metrics can form the basis of measuring psychometric principles that ECD uses to evaluate assessments, which are validity, reliability, comparability and fairness (Mislevy & Wilson, 2003).

The interplay of game elements with psychometric qualities, learning, and enjoyment in game-based assessment

Educators today are increasingly interested in using game-based assessment to assess and support students' learning. In the present study, we investigated how changing a game design element, linearity in gameplay sequences, influenced the effectiveness of game-based assessment in terms of validity, reliability, fairness, learning, and enjoyment. Two versions of a computer game, Physics Playground (formerly Newton's Playground), with different degrees of linearity in gameplay sequences were compared. Investigation of the assessment qualities—validity, reliability, and fairness—suggested that changing one game element (e.g., linearity) could significantly influence how players interacted with the game, thus changing the evidentiary structure of in-game measures. Although there was no significant group difference in terms of learning between the two conditions, participants who played the nonlinear version of the game showed significant improvement on qualitative physics understanding measured by the pre- and posttests while the participants in the linear condition did not. There was also no significant group difference in terms of enjoyment. Implications of the findings for future researchers and game-based assessment designers are discussed.

Posterlet: A Game-Based Assessment of Children's Choices to Seek Feedback and to Revise

We introduce one instance of a game-based assessment designed to measure students’ self-regulated learning choices. We describe our overarching measurement strategy and we present Posterlet, an assessment game in which students design posters and learn graphic design principles from feedback. We designed Posterlet to assess children’s choices to seek informative negative feedback and to revise their work. Middle-school students from New York and Illinois played Posterlet and then took a post-test, for an overall average of 17 minutes of interaction time. Results showed that the frequency of choosing negative feedback and revision correlated with learning graphic design principles from the game. Seeking negative feedback, but less so revision, further correlated with standardized achievement tests of reading and mathematics. Our research presents a first-of-kind behavioural measure of students’ feedback and revision choices and their relations to learning. Within the design context of creating posters, we found correlational evidence that seeking negative feedback and revising are good behaviours for self-regulated learning, and we devised a way to measure these behaviours. This sets the stage for developing and evaluating models of self-regulated learning instruction that help students choose to seek feedback effectively and revise accordingly.

Differences Between Japanese and U.S. Children's Performance on "Zoo U": A Game-Based Social Skills Assessment.

The purpose of this study was to test whether and how performance on a digital game-based social skills assessment tool, "Zoo U" (3C Institute, Durham, NC), differed for children in the United States and Japan across six core social skills. "Zoo U" was administered to 497 third and fourth grade children from the United States and Japan (46 percent Japanese) by teachers and researchers, respectively. U.S. children received the original version of "Zoo U," and Japanese children received a fully translated Japanese version of the program. Scoring of each of the six social skills is built into the "Zoo U" software, with specific scoring algorithms for each grade level that provide both a continuous scale score and cutoffs for three distinct performance categories: high, average, and low. A multivariate analysis of variance (MANOVA) was conducted to assess differences by cultural group on each of the six continuous social skill scores, controlling for gender and grade level. Results revealed significant differences on four of the six skills in expected directions. Chi-squared and odds ratios analyses were then conducted on the assignment of children into each of the performance categories by cultural group, revealing additional nuance to the cultural differences identified in the MANOVA consistent with existing literature. We were able to replicate known cultural differences between U.S. and Japanese children with a simple direct translation of a Web-based social skills assessment game, "Zoo U." Our results provide preliminary support for the potential of game-based assessment methods to provide efficient and valid social skill assessments to children around the world.

Psychometric analysis of the performance data of simulation-based assessment: A systematic review and a Bayesian network example

Researchers have shown in multiple studies that simulations and games can be effective and powerful tools for learning and instruction (cf. Mitchell & Savill-Smith, 2004; Kirriemuir & McFarlane, 2004). Most of these studies deploy a traditional pretest-posttest design in which students usually do a paper-based test (pretest) then play the simulation or game and subsequently do a second paper-based test (posttest). Pretest-posttest designs treat the game as a black box in which something occurs that influences subsequent performance on the posttest (Buckley, Gobert, Horwitz, & O'Dwyer, 2010). Less research has been done in which game play product data or process data itself are used as indicators of student proficiency in some area. However, the last decade researchers have started focusing on what is happening inside the black box to an increasing extent and the literature on the topic is growing. To our knowledge, no systematic reviews have been published that investigate the psychometric analysis of performance data of simulation-based assessment (SBA) and game-based assessment (GBA). Therefore, in Part I of this article, a systematic review on the psychometric analysis of the performance data of SBA is presented. The main question addressed in this review is: ‘What psychometric strategies or models for treating and analyzing performance data from simulations and games are documented in scientific literature?’. Then, in Part II of this article, the findings of our review are further illustrated by presenting an empirical example of the – according to our review – most applied psychometric model for the analysis of the performance data of SBA, which is the Bayesian network. Both the results from Part I and Part II assist future research into the use of simulations and games as assessment instruments.

Game-based assessment: an integrated model for capturing evidence of learning in play

This paper presents the assessment data aggregator for game environments (ADAGE), a click-stream data framework designed to test whether click-stream data can provide reliable evidence of learning. As digitally-based games become increasingly prevalent in both formal and informal learning environments, robust research and assessment vehicles within well-designed games become vital. A central challenge with this video games research in education is to demonstrate evidence of player learning. Rather than ignore the motivating and information-rich features of games in capturing learning, assessment designers need to attend to the ways in which game-play itself can provide a powerful new form of assessment. This requires learning researchers to think of games as both intervention and assessment; and to develop methods for using the internal structures of games as paths for evidence generation to document learning. ADAGE consists of two main layers: 1) the semantic template that determines which click-stream data events could be indicators of learning; 2) the learning telemetry that captures data for analysis. This study highlights how ADAGE was implemented in science game to provide a wealth of in-game player data indicative of both gameplay and learning progress to reveal important relationships between kinds of success, failure and learning in the game.

Visual signaling in virtual world-based assessments: The SAVE Science project

In this paper we describe a study into the impact of visual signaling techniques used in a virtual world-based assessment of science inquiry and content on (1) student cognitive load and (2) assessment efficiency. The study, run with 7th grade students in the United States, found that use of visual signaling was significantly associated with lower levels of student self-reported cognitive load versus students in a no-signaling version of the assessment. Further, the efficiency of the virtual world-based assessment was significantly higher, as measured by in-world object interaction rates, for students in the visual signaling version of the assessment than for those in the no-signaling treatment. In the paper, we discuss the results and their meaning for the design of virtual world and game-based assessments.

The validity of a game-based assessment of persistence

In this study, 154 students individually played a challenging physics video game for roughly 4h. Based on time data for both solved and unsolved problems derived from log files, we created a game-based assessment of persistence that was validated against an existing measure of persistence. We found that the game-based assessment of persistence predicted learning of qualitative physics after controlling for gender, video game experience, pretest knowledge and enjoyment of the game. These findings support the implementation of a real-time formative assessment of persistence to be used to dynamically change gameplay.

Game-based assessment of first aid and resuscitation skills

Aim Effective assessment of motor skills in large-size classes is a challenge in medical education. This case-study investigates whether a game can be considered a valid tool for the summative assessment of first aid and basic life support skills. Methods Using a traditional exam as bench-mark, a board game format was experimentally trialed to assess students’ competency after taking a first aid course. Fifty-five students were randomly assigned to two groups. Two assessments, a game-based assessment and a traditional test, consisting of a paper-and-pencil test in combination with a skill assessment, were applied to both groups in opposite order. In both formats students acted as judges of other students’ efforts. In the game, the student's outcome was equal to the number of cards collected by answering questions correctly as deemed by peers. Similarities between both assessment types included individual testing, type of assessor (peers), content, type of questions and demonstrations, and the use of checklists for skill assessment. The assessment methods differed in format (written or oral test, both in combination with skill assessment) and feedback availability. Results Both groups performed equally in the game-based assessment as well as in the traditional test, in spite of the opposite order of the assessments. No significant difference was found between the mean scores on the game-based assessment and the traditional test. Conclusions These data suggest that use of a game format for assessment purposes may provide an effective means of assessing students’ competence at the end of a practical course.

Preschoolers' number sense

Consider using a game-based assessment of number sense in young children, including those at risk because of socioeconomic level, disability, or the necessity of learning a second language.

Development and Evaluation of the Game-Based Performance Assessment Battery (GamePAB) and Game Experience Measure (GEM)

Development and Evaluation of the Game-Based Performance Assessment Battery (GamePAB) and Game Experience Measure (GEM)