Complex Video Game Research Articles

Towards Human-Level Safe Reinforcement Learning in Atari Library

Reinforcement learning (RL) is a powerful tool for training agents to perform complex tasks. However, from time-to-time RL agents often learn to behave in unsafe or unintended ways. This is especially true during the exploration phase, when the agent is trying to learn about its environment. This research acquires safe exploration methods from the field of robotics and evaluates their effectiveness compared to other algorithms that are commonly used in complex videogame environments without safe exploration. We also propose a method for hand-crafting catastrophic states, which are states that are known to be unsafe for the agent to visit. Our results show that our method and our hand-crafted safety constraints outperform state-of-the-art algorithms on relatively certain iterations. This means that our method is able to learn to behave safely while still achieving good performance. These results have implications for the future development of human-level safe learning with combination of model-based RL using complex videogame environments. By developing safe exploration methods, we can help to ensure that RL agents can be used in a variety of real-world applications, such as self-driving cars and robotics.

Self-orienting in human and machine learning.

A current proposal for a computational notion of self is a representation of one's body in a specific time and place, which includes the recognition of that representation as the agent. This turns self-representation into a process of self-orientation, a challenging computational problem for any human-like agent. Here, to examine this process, we created several 'self-finding' tasks based on simple video games, in which players (N = 124) had to identify themselves out of a set of candidates in order to play effectively. Quantitative and qualitative testing showed that human players are nearly optimal at self-orienting. In contrast, well-known deep reinforcement learning algorithms, which excel at learning much more complex video games, are far from optimal. We suggest that self-orienting allows humans to flexibly navigate new settings.

Event-related (de)synchronization and potential in whole vs. part sensorimotor learning.

There are different ways to learn a sensorimotor task. This research focuses on whole versus part learning in a complex video game that involves sensorimotor adaptations and skill learning. The primary aim of this research is to compare the changes in (1) event-related potentials (ERP) and (2) Alpha and Beta event-related desynchronization/synchronization [ERD(S)] of EEG between whole and part practice protocols. 18 Healthy young participants practiced for 5 days a video game with distorted kinematic (advancing skill) and dynamic features (shooting skill) to test the ability to combine sensorimotor skill components learned modularly (part learning, 9 participants) or combined (whole practice, 9 participants). We examined ERP and ERD(S) in EEG channels in the baseline test (day 1) and the retention test (day 5), dissociating epochs with advancing or shooting. We focus the analysis on the main activity of ERP or ERD(S) in different time windows. In the advancing epochs (distorted kinematic), both groups showed a decrease in time for ERP and an increase in Beta ERD activity in central and posterior channels. In the shooting epochs (distorted dynamic), the Whole group showed a decrease in time for ERPs in anterior and central-posterior channels. Additionally, the shooting ERS in the Beta band decreases within sessions in central channels, particularly for the Part group. Neural correlates of kinematic and dynamic control [ERP and ERD(S)] were modulated by sensorimotor learning, which reflects the effect of the type of practice on the execution and the evaluation of the action. These results can be linked with our previous report, where the simultaneous practice of kinematic and dynamic distortions takes advantage of the motor performance on retention tests, indicating a more automatic control for the whole practice group.

Figures of the avatar: semi-autonomous agents, presence effects and interpellation in videogames

In this article, we analyse the strategies that allow non-player characters to create the illusion that they are autonomous agents in the service of complex videogame worlds. We describe these characters as semi-autonomous agents, and approach them as mediation devices that can induce a presence effect and “interpellate” the players’ avatars in order to make them more emotionally present and involved in the videogame world. We proceed in two stages. First, we examine the immersive and simulative logics that foster the impression that these agents are part of an autonomous world, almost as though videogame worlds didn’t rely on the presence of the players’ avatars. Second, we probe these logics further using Althusser’s analysis of interpellation in order to approach videogame worlds as simulating both worlds and relationships to the world. We consider semi-autonomous agents as sophisticated “interpellators” that have been carefully programmed to capture the players’ attention and assign them a fixed role or function. However, interpellation is not a one-way way process because it fosters “counter-interpellations,” involving the players in the game, not just on the level of their performance, but also as subjects in the videogame world within which they feel they exist. We use concrete examples drawn from a comparative study of the interpellation and presence effect mechanisms that we have identified in two single-player videogames, namely Red Dead Redemption 2 (2018) and The Last of Us 2 (2020), both of which have gained recognition for their character design and for the quality and depth of their diegetic universes.

Neuroanatomical predictors of complex skill acquisition during video game training.

It is known that the outcomes of complex video game (VG) skill acquisition are correlated with individual differences in demographic and behavioral variables, such as age, intelligence and visual attention. However, empirical studies of the relationship between neuroanatomical features and success in VG training have been few and far between. The present review summarizes existing literature on gray matter (GM) and white matter correlates of complex VG skill acquisition as well as explores its relationship with neuroplasticity. In particular, since age can be an important factor in the acquisition of new cognitive skills, we present studies that compare different age groups (young and old adults). Our review reveals that GM in subcortical brain areas predicts complex VG learning outcomes in young subjects, whereas in older subjects the same is true of cortical frontal areas. This may be linked to age-related compensatory mechanisms in the frontal areas, as proposed by The Scaffolding Theory of Aging and Cognition. In the case of plasticity, there is no such relationship – in the group of younger and older adults there are changes after training in both cortical and subcortical areas. We also summarize best practices in research on predictors of VG training performance and outline promising areas of research in the study of complex video game skill acquisition.

Novel 3-D action video game mechanics reveal differentiable cognitive constructs in young players, but not in old

Video game research predominantly uses a “one game-one function” approach—researchers deploy a constellation of task-like minigames to span multiple domains or consider a complex video game to essentially represent one cognitive construct. To profile cognitive functioning in a more ecologically valid setting, we developed a novel 3-D action shooter video game explicitly designed to engage multiple cognitive domains. We compared gameplay data with results from a web-based cognitive battery (WebCNP) for 158 participants (aged 18–74). There were significant negative main effects on game performance from age and gender, even when controlling for prior video game exposure. Among younger players, game mechanics displayed significant and unique correlations to cognitive constructs such as aim accuracy with attention and stealth with abstract thinking within the same session. Among older players the relation between game components and cognitive domains was unclear. Findings suggest that while game mechanics within a single game can be deconstructed to correspond to existing cognitive metrics, how game mechanics are understood and utilized likely differs between the young and old. We argue that while complex games can be utilized to measure distinct cognitive functions, the translation scheme of gameplay to cognitive function should not be one-size-fits-all across all demographics.

Behavioral Cues of Humanness in Complex Environments: How People Engage With Human and Artificially Intelligent Agents in a Multiplayer Videogame.

The development of AI that can socially engage with humans is exciting to imagine, but such advanced algorithms might prove harmful if people are no longer able to detect when they are interacting with non-humans in online environments. Because we cannot fully predict how socially intelligent AI will be applied, it is important to conduct research into how sensitive humans are to behaviors of humans compared to those produced by AI. This paper presents results from a behavioral Turing Test, in which participants interacted with a human, or a simple or “social” AI within a complex videogame environment. Participants (66 total) played an open world, interactive videogame with one of these co-players and were instructed that they could interact non-verbally however they desired for 30 min, after which time they would indicate their beliefs about the agent, including three Likert measures of how much participants trusted and liked the co-player, the extent to which they perceived them as a “real person,” and an interview about the overall perception and what cues participants used to determine humanness. T-tests, Analysis of Variance and Tukey's HSD was used to analyze quantitative data, and Cohen's Kappa and χ2 was used to analyze interview data. Our results suggest that it was difficult for participants to distinguish between humans and the social AI on the basis of behavior. An analysis of in-game behaviors, survey data and qualitative responses suggest that participants associated engagement in social interactions with humanness within the game.

Emotion-Performance Relationships in the Acquisition and Adaptation of a Complex Skill: Are Relationships Dynamic and Dependent on Activation Potential?

ABSTRACT Although the importance of emotions to self-regulation has been noted in the extant literature, little empirical research has examined how fluctuations across a range of emotions are related to performance. Accordingly, this laboratory study of 214 undergraduates (58% male) learning a complex videogame examined the dynamics of within-person emotion–performance relationships during skill acquisition and adaptation. We observed the direction of relationships was consistent with the valence of emotions at both the within- and between-person levels. In addition, we observed dynamic relationships and an interplay between valence and activation potential. Dynamic relationships were observed for negative but not positive emotions. Activation potential made a difference for positive but not negative emotions. Implications for emotion control strategies in complex task learning are discussed.

Keeping Calm and Carrying On: Relating Affect Spin and Pulse to Complex Skill Acquisition and Adaptive Performance.

The purpose of this laboratory study involving repeated measures of emotion as 214 undergraduates (58.4% male) learned a complex video game was to address the need for empirical research on dynamic personality constructs by examining how two aspects of affect variability—spin and pulse—explain variance in skill acquisition and adaptive performance. Spin refers to within-person fluctuations in affect pleasantness and activation potential. Pulse refers to within-person fluctuations in affect intensity. Despite research showing high affect variability reflects a personality profile of heighted reactivity to emotionally charged events and poor adjustment, little empirical research has examined their relationships with behavioral outcomes, much less aspects of skilled performance. Compared to traditional measures of personality, which yield weak effects for predicting acquisition and adaptive performance, measures of affect variability hold considerable promise because they, like performance, reflect dynamic within-person phenomena. Accordingly, the main question addressed by this study was whether spin and pulse incrementally explain acquisition and adaptive performance beyond Big Five measures of personality. In general, we expected harmful, incremental effects for both spin and pulse, and hypothesized two mechanisms for these harmful effects: (1) by undermining effort and (2) by undermining the effort-performance relationship. Using a task-change paradigm and discontinuous growth modeling that disentangled adaptation from acquisition, results showed that affect variability, independent of the Big Five, produced harmful effects via both hypothesized mechanisms. Participants higher in affect spin and pulse showed less sustained effort across performance sessions and exhibited lower performance. Furthermore, the harmful effects of spin and pulse were stronger in adaptation compared to acquisition, with pulse showing stronger direct effects on performance during adaptation and spin moderating the effort-performance relationship such that effort was only beneficial during adaptation for those lower in spin. In light of these results, one might question the common advice “keep calm and carry on,” which may not be viable for persons high in affect variability. Accordingly, results are discussed in terms of the need to better understand the specific mediating processes by which high affect variability undermines success across a variety of learning and performance contexts.

Location, Integration, Interruption: Visual Properties and Recognition of Video Game Advertising

Videogame placements are significant forms of advertising. There is, however, a limited amount of cognitive capacity available to process messages at any given moment during a complex video game. Using the Limited Capacity Model as an exploratory mechanism, it is proposed that the recall and recognition of in-game advertising depends on its positioning in the field-of-view and its integration within the game world. This is a quasi-experimental study involving 42 Arab women, key purchasing decision-makers and high-level consumers of luxury products, playing a commercial driving videogame. Participants were questioned about message recognition after 40 minutes of play. Placement positions in the center of the screen enhanced brand and message recognition. Ads integrated into the game environment were better recognized than virtual billboards. Landmarks were better recognized than either integrated ads or billboards.

The Social Facilitation of Performance, Engagement and Affect in a Complex Videogame

The objective of this study is to analyze the social facilitation of performance, intrinsic engagement, state hostility, and targeted affects in a computer-based driving game where social actors are competitors. This is a quasi-experimental study with 97 Gulf Arab women. Social facilitation of absolute performance does not take place while it does for relative performance. There is no difference in state hostility based on social facilitation, but there is in targeted affect. Intrinsic engagement and extrinsic motivation are both facilitated by human opponents. There is a negative relationship between intrinsic engagement and state hostility across conditions. There is evidence that the experience of playing a game character and playing a person is substantially different. The two most powerful predictors of performance and affect are intrinsic engagement and videogame interest when playing a person. Weekly hours of console play are added to those two when playing a game character.

Pilot Study of Mental Flexibility brain networks

Pilot Study of Mental Flexibility brain networks

The sequential structure of brain activation predicts skill

In an fMRI study, participants were trained to play a complex video game. They were scanned early and then again after substantial practice. While better players showed greater activation in one region (right dorsal striatum) their relative skill was better diagnosed by considering the sequential structure of whole brain activation. Using a cognitive model that played this game, we extracted a characterization of the mental states that are involved in playing a game and the statistical structure of the transitions among these states. There was a strong correspondence between this measure of sequential structure and the skill of different players. Using multi-voxel pattern analysis, it was possible to recognize, with relatively high accuracy, the cognitive states participants were in during particular scans. We used the sequential structure of these activation-recognized states to predict the skill of individual players. These findings indicate that important features about information-processing strategies can be identified from a model-based analysis of the sequential structure of brain activation.

The relationship between intelligence and training gains is moderated by training strategy.

We examined the relationship between training regimen and fluid intelligence in the learning of a complex video game. Fifty non-game-playing young adults were trained on a game called Space Fortress for 30 hours with one of two training regimens: 1) Hybrid Variable-Priority Training (HVT), with part-task training and a focus on improving specific skills and managing task priorities, and 2) Full Emphasis Training (FET) in which participants practiced the whole game to obtain the highest overall score. Fluid intelligence was measured with the Raven’s Progressive Matrix task before training. With FET, fluid intelligence was positively associated with learning, suggesting that intellectual ability played a substantial role in determining individual differences in training success. In contrast, with HVT, fluid intelligence was not associated with learning, suggesting that individual differences in fluid intelligence do not factor into training success in a regimen that emphasizes component tasks and flexible task coordination. By analyzing training effects in terms of individual differences and training regimens, the current study offers a training approach that minimizes the potentially limiting effect of individual differences.

The challenges of developing a contrast-based video game for treatment of amblyopia.

Perceptual learning of visual tasks is emerging as a promising treatment for amblyopia, a developmental disorder of vision characterized by poor monocular visual acuity. The tasks tested thus far span the gamut from basic psychophysical discriminations to visually complex video games. One end of the spectrum offers precise control over stimulus parameters, whilst the other delivers the benefits of motivation and reward that sustain practice over long periods. Here, we combined the advantages of both approaches by developing a video game that trains contrast sensitivity, which in psychophysical experiments, is associated with significant improvements in visual acuity in amblyopia. Target contrast was varied adaptively in the game to derive a contrast threshold for each session. We tested the game on 20 amblyopic subjects (10 children and 10 adults), who played at home using their amblyopic eye for an average of 37 sessions (approximately 11 h). Contrast thresholds from the game improved reliably for adults but not for children. However, logMAR acuity improved for both groups (mean = 1.3 lines; range = 0–3.6 lines). We present the rationale leading to the development of the game and describe the challenges of incorporating psychophysical methods into game-like settings.

Parietal plasticity after training with a complex video game is associated with individual differences in improvements in an untrained working memory task

Researchers have devoted considerable attention and resources to cognitive training, yet there have been few examinations of the relationship between individual differences in patterns of brain activity during the training task and training benefits on untrained tasks (i.e., transfer). While a predominant hypothesis suggests that training will transfer if there is training-induced plasticity in brain regions important for the untrained task, this theory lacks sufficient empirical support. To address this issue we investigated the relationship between individual differences in training-induced changes in brain activity during a cognitive training videogame, and whether those changes explained individual differences in the resulting changes in performance in untrained tasks. Forty-five young adults trained with a videogame that challenges working memory, attention, and motor control for 15 2-h sessions. Before and after training, all subjects received neuropsychological assessments targeting working memory, attention, and procedural learning to assess transfer. Subjects also underwent pre- and post-functional magnetic resonance imaging (fMRI) scans while they played the training videogame to assess how these patterns of brain activity change in response to training. For regions implicated in working memory, such as the superior parietal lobe (SPL), individual differences in the post-minus-pre changes in activation predicted performance changes in an untrained working memory task. These findings suggest that training-induced plasticity in the functional representation of a training task may play a role in individual differences in transfer. Our data support and extend previous literature that has examined the association between training related cognitive changes and associated changes in underlying neural networks. We discuss the role of individual differences in brain function in training generalizability and make suggestions for future cognitive training research.

The Influence of Single-gender Peer Scaffolding in Problem-based Gaming on Performance in Double-loop Learning and Sub-dimensions of Science Process Skills

This study investigated gender differences in problem-based gaming with peer scaffolding on engagement in double-loop learning (DLL) and performance in the sub-dimensions of science process skills, namely, observing, classifying, inferring, predicting, controlling variables, making hypothesis and experimenting. A complex video game was chosen and a quasi- experimental study involving 60 fifth grade students in two groups of single-gender peer scaffolding and individual was employed. Overall the peer scaffolding group engaged in significantly more DLL cycles than the individual group. Further analysis found that boys in both the individual and single-gender groups engaged in significantly more DLL cycles than girls. Also, single-gender peer scaffolding significantly increased engagement in DLL for girls and performance in the sub-dimension of experimenting for boys.

Reinforcement learning agents providing advice in complex video games

This article introduces a teacher–student framework for reinforcement learning, synthesising and extending material that appeared in conference proceedings [Torrey, L., & Taylor, M. E. (2013)]. Teaching on a budget: Agents advising agents in reinforcement learning. {Proceedings of the international conference on autonomous agents and multiagent systems}] and in a non-archival workshop paper [Carboni, N., &Taylor, M. E. (2013, May)]. Preliminary results for 1 vs. 1 tactics in StarCraft. {Proceedings of the adaptive and learning agents workshop (at AAMAS-13)}]. In this framework, a teacher agent instructs a student agent by suggesting actions the student should take as it learns. However, the teacher may only give such advice a limited number of times. We present several novel algorithms that teachers can use to budget their advice effectively, and we evaluate them in two complex video games: StarCraft and Pac-Man. Our results show that the same amount of advice, given at different moments, can have different effects on student learning, and that teachers can significantly affect student learning even when students use different learning methods and state representations.

Cognitive Ability Predicts Older Adult Performance in a Complex Task but is Moderated by Social Interaction

Previous research has examined the effects of cognitive training on cognitive abilities. However, the contribution of cognitive abilities to learning to perform the tasks used as cognitive training, such as complex video games, is not completely understood. Additionally, social interaction while playing a video game may change the relationship between cognitive ability and game success by allowing players to learn from one another instead of solely their own abilities. As part of a larger cognitive intervention designed for older adults, the 50 participants included in our analysis were randomly assigned to either play games alone or take turns with a partner for one hour a day over fifteen days. Those playing with a partner outscored those playing alone. Multiple cognitive abilities predicted game success, but there was also evidence to suggest a moderating effect when playing with a partner on the predictive power of certain cognitive abilities.

Learner-controlled practice difficulty in the training of a complex task: Cognitive and motivational mechanisms.

An inherent aspect of learner-controlled instructional environments is the ability of learners to affect the degree of difficulty faced during training. However, research has yet to examine how learner-controlled practice difficulty affects learning. Based on the notion of desirable difficulties (Bjork, 1994), this study examined the cognitive and motivational antecedents and outcomes of learner-controlled practice difficulty in relation to learning a complex task. Using a complex videogame involving both strong cognitive and psychomotor demands, 112 young adult males were given control over their practice difficulty, which was reflected in the complexity of the training task. Results show that general mental ability, prior experience, pre-training self-efficacy, and error encouragement were positively related to learner-controlled practice difficulty. In turn, practice difficulty was directly related to task knowledge and post-training performance, and it was related to adaptive performance through the mediating influences of task knowledge and post-training performance. In general, this study supports the notion that training difficulty operationalized in terms of task complexity is positively related to both knowledge and performance outcomes. Results are discussed with respect to the need for more research examining how task complexity and other forms of difficulty could be leveraged to advance learner-controlled instructional practices.