A New Deep Learning Fusion Approach for Emotion Recognition Based on Face and Text

Abstract

Automatic emotion recognition has attracted much interest in the last years and is becoming a challenging task. One modality by itself does not carry all the information to convey and perceive human emotions. Also, sometimes, it isn’t easy to choose between several affective states. To remove these ambiguities, we propose a deep learning-based decision-level fusion approach for Facial Textual Emotion Recognition (FTxER) to classify emotions into discrete emotion classes. Our approach is based on Deep Convolution Neural Network (DCNN) and Bidirectional Long Short Term Memory (BiLSTM). We use the latter to improve the correlation of the time dimension of DCNN face data. Our experiments on the CK+ dataset show that the weighted average of F1-score of the FTxER model is about $$79\%$$ .