Affective state detection via facial expression analysis within a human\u2013computer interaction context

Anas Samara,Raymond Bond,Leo Galway,Hui Wang

doi:10.1007/s12652-017-0636-8

Anas Samara, Raymond Bond + Show 2 more

Open Access

https://doi.org/10.1007/s12652-017-0636-8

Copy DOI

Abstract

The advancement in technology indicates that there is an opportunity to enhance human–computer interaction by way of affective state recognition. Affective state recognition is typically based on passive stimuli such as watching video clips, which does not reflect genuine interaction. This paper presents a study on affective state recognition using active stimuli, i.e. facial expressions of users when they attempt computerised tasks, particularly across typical usage of computer systems. A data collection experiment is presented for acquiring data from normal users whilst they interact with software, attempting to complete a set of predefined tasks. In addition, a hierarchical machine learning approach is presented for facial expression-based affective state recognition, which employs an Euclidean distance-based feature representation, conjointly with a customised encoding for users’ self-reported affective states. Consequently, the aim is to find the potential relationship between the facial expressions, as defined by Paul Ekman, and the self-reported emotional states specified by users using Russells Circumplex model, in relation to the actual feelings and affective states. The main findings of this study suggest that facial expressions cannot precisely reveal the actual feelings of users whilst interacting with common computerised tasks. Moreover, during active interaction tasks more variation occurs within the facial expressions of participants than occurs within passive interaction.

Highlights

In human-human interaction, one can intuitively predict the emotional state based on observations about persons facial expressions, body behaviour, and voice intonations (Karray et al 2008)
Percentages of facial expression that automatically applied on the video frames from the dataset obtained from the current work, which are recordings of subjects interacting with typical software interfaces
Percentages of facial expressions across self-reporting ratings given by subjects, that are presumed to represent their actual feeling during each task, as well as, facial expression percentages according to combined ratings that are mapped to the Circumplex model quadrants

Summary

Introduction

In human-human interaction, one can intuitively predict the emotional state based on observations about persons facial expressions, body behaviour, and voice intonations (Karray et al 2008). This model is composed of valence and arousal intensity dimensions, whereby valence represents the intrinsic attractiveness or averseness of an emotion, and can be presented as a pleasantunpleasant continuum (Frijda 1986), and arousal is the physiological and psychological state that activates the alertness, consciousness and attention as a reaction to stimuli, and can be presented as an activation-deactivation continuum (Coull 1998; Robbins 1997). The work presented may be considered as comparable in terms of facial expression classification accuracy, whereas this work has been validated against multiple benchmark datasets

Objectives

Methods

Results

Conclusion