FaceEngage: Robust Estimation of Gameplay Engagement from User-Contributed (YouTube) Videos

Xu Chen,Ashutosh Sabharwal,Li Niu,Ashok Veeraraghavan

doi:10.1109/taffc.2019.2945014

Xu Chen, Ashutosh Sabharwal + Show 2 more

Open Access

https://doi.org/10.1109/taffc.2019.2945014

Copy DOI

Abstract

Measuring user engagement in interactive tasks can facilitate numerous applications toward optimizing user experience, ranging from eLearning to gaming. However, a significant challenge is the lack of non-contact engagement estimation methods that are robust in unconstrained environments. We present FaceEngage, a non-intrusive engagement estimator leveraging user facial recordings during actual gameplay in naturalistic conditions. Our contributions are three-fold. First, we show the potential of using front-facing videos as training data to build the engagement estimator. We compile FaceEngage Dataset with over 700 picture-in-picture, realisitic, and user-contributed YouTube gaming videos (i.e., with both full-screen game scenes and time-synchronized user facial recordings in subwindows). Second, we develop FaceEngage system, that captures relevant gamer facial features from front-facing recordings to infer task engagement. We implement two FaceEngage pipelines: an estimator trained on user facial motion features inspired by prior psychological works, and a deep learning-enabled estimator. Lastly, we conduct extensive experiments and conclude: (i) certain user facial motion cues (e.g., blink rates, head movements) are engagement-indicative; (ii) our deep learning-enabled FaceEngage pipeline can automatically extract more informative features, outperforming the facial motion feature-based pipeline; (iii) FaceEngage is robust to various video lengths, users/game genres and interpretable. Despite the challenging nature of realistic videos, FaceEngage attains the accuracy of 83.8 percent and leave-one-user-out precision of 79.9 percent, both of which are superior to our face motion-based model.

Highlights

Measuring user cognitive states, such as task engagement, is desirable in many interactive applications e.g., to gauge user engagement during eLearning, gaming and digital media consumption
We present a supervised learning-based FaceEngage algorithm, which is trained to capture informative users’ facial features for inferring their gameplay engagement levels, which achieves the accuracy of 83.8%
Through extensive experiments on FaceEngage algorithms, we find that (i) motion cues on users’ faces, such as eye-blink rates, gaze and head pose movements are indicative of their engagement levels, among which the blink rate is the most indicative feature; (ii) the FaceEngage algorithm can well-generalize to new users and games; and (iii) the FaceEngage algorithm is robust to various input video lengths, user appearances and game genres with interpretability

Summary

Introduction

Measuring user cognitive states, such as task engagement, is desirable in many interactive applications e.g., to gauge user engagement during eLearning, gaming and digital media consumption. Estimating user engagement continues to be challenging as it is an internal cognitive state with few visible clues. In the first category of self-report based measures, participants engage in a task and are asked on their self-perceived engagement level. Self-reported measures are easier to use but face a significant sampling challenge. To get sufficient time samples during the course of the activity, (in some experiments) the participants are asked to pause their ongoing activity, asked to report on their self-perceived

Methods

Results

Conclusion