Multimodal human attention detection for reading from facial expression, eye gaze, and mouse dynamics

Abstract

Affective computing has recently become an important area in human-computer interaction research. Techniques have been developed to enable computers to understand human affects or emotions, in order to predict human intention more precisely and provide better service to user to enhance user experience. In this paper, we investigate into the detection of human attention level as a useful from of human affect, which could be influential in intelligent e-learning applications. We adopt ubiquitous hardware available in most computer systems, namely, webcam and mouse. Information from multiple input modalities is fused together for effective human attention detection. We invite human subjects to carry out experiments in reading articles when being imposed upon different kinds of distraction to induce them into different levels of attention. Machine-learning techniques are applied to identify useful features to recognize human attention level by building up user-independent models. Our result indicate performance improvement with multimodal inputs from webcam and mouse over that of a single device. We believe that our work has revealed and intersting affective computing direction with potential application in e-learning.