Dual-modality spatiotemporal feature learning for spontaneous facial expression recognition in e-learning using hybrid deep neural network

Abstract

Automatic facial expression recognition (FER) plays a crucial role in realizing the adaptable and individualized tutoring in affective computer-based learning environment. Although many research efforts have been conducted to enhance a greater understanding of FER, a successful accurate recognition of the spontaneous facial expressions in real e-learning environment is still challenging due to its low change in intensity and short duration. In this paper, we propose a new dual-modality spatiotemporal feature representation learning for recognizing facial expression in e-learning using the hybrid deep neural network. Except facial expression class information, representative expression states (e.g., onset, apex, offset of expressions) are utilized for expression recognition in our study. Spatiotemporal geometrical feature representations and spatial-temporal appearance feature representations are learned with a hybrid deep neural network. The dual-modality feature fusion representations are used to recognize facial expressions. The comprehensive experiments have been conducted on two spontaneous micro-expression datasets ($$\hbox {CAS(ME)}^2$$ and CASME II). The experimental results showed that the proposed method achieved higher recognition accuracy compared to the state-of-the-art methods. Moreover, multiple metrics were adopted to provide more insight into the performance of the proposed method.