A Human–Computer Interaction framework for emotion recognition through time-series thermal video sequences

Abstract

Infrared-Thermal Imaging is a non-contact mechanism for psychophysiological research and application in Human–Computer Interaction (HCI). Real-time detection of the face and tracking the Regions of Interest (ROI) in the thermal video during HCI is challenging due to head motion artifacts. This paper proposes a three-stage HCI framework for computing the multivariate time-series thermal video sequences to recognize human emotion and provides distraction suggestions. The first stage comprises of face, eye, and nose detection using a Faster R-CNN (region-based convolutional neural network) architecture and used Multiple Instance Learning (MIL) algorithm for tracking the face ROIs across the thermal video. The mean intensity of ROIs is calculated which forms a multivariate time series (MTS) data. In the second stage, the smoothed MTS data are passed to the Dynamic Time Warping (DTW) algorithm to classify emotional states elicited by video stimulus. During HCI, our proposed framework provides relevant suggestions from a psychological and physical distraction perspective in the third stage. Our proposed approach signifies better accuracy in comparison with other classification methods and thermal data-sets.