Emotion recognition with facial expressions and physiological signals

Abstract

This paper proposes a temporal information preserving multi-modal emotion recognition framework based on physiological and facial expression data streams. The performance of each component is evaluated and compared individually and after data fusion. Specifically, we compared the effect of different views of cameras on facial expressions for emotion recognition, and combined these views to achieve better performance. A Temporal Information Preserving Framework (TIPF) is proposed to more accurately model the relationships between emotional and physiological states over time. Additionally, different fusion strategies are compared when combining information from different time periods and modalities. The experiments show that, TIPF significantly improves the emotion recognition performance when physiological signals are used and the best performance is achieved when fusing facial expressions and physiological data.