Towards gaze-controlled platform games

Abstract

This paper introduces the concept of using gaze as a sole modality for fully controlling player characters of fast-paced action computer games. A user experiment is devised to collect gaze and gameplay data from subjects playing a version of the popular Super Mario Bros platform game. The initial analysis shows that there is a rather limited grid around Mario where the efficient player focuses her attention the most while playing the game. The useful grid as we name it, projects the amount of meaningful visual information a designer should use towards creating successful player character controllers with the use of artificial intelligence for a platform game like Super Mario. Information about the eyes' position on the screen and the state of the game are utilized as inputs of an artificial neural network, which is trained to approximate which keyboard action is to be performed at each game step. Results yield a prediction accuracy of over 83% on unseen data samples and show promise towards the development of eye-controlled fast-paced platform games. Derived neural network players are intended to be used as assistive technology tools for the digital entertainment of people with motor disabilities.