EmoSeC: Emotion recognition from scene context

Abstract

Context provides additional information to determine the actual emotional state of a person as part of a scene. Existing works on emotion recognition in context focused only on the features extracted from the entire image and the target’s body. In this work, we propose a comprehensive multi-cue based emotion recognition framework that incorporates the context, using a hybrid architecture comprised of four separate deep convolutional neural networks and a novel feature fusion mechanism. Each deep network presented in our proposed approach effectively learns the emotion-related features from the facial, body pose, non-target subject and entire image, individually. Experiments on Emotic, an in-painted and a newly constructed EmoSec datasets show that our proposed emotion recognition framework is promising when compared to existing methods in terms of accurately classifying emotions. Comparison with the state-of-the-art deep networks and off-the-shelf fusion techniques demonstrates that our network showed an improved performance. Furthermore, the performance evaluation of our proposed approach on all three datasets confirms that the contextual information influences the emotional state of a human.