Concordance between facial micro-expressions and physiological signals under emotion elicitation

Abstract

• Synchronized collection of micro-expressions and peripheral physiological signals. • Micro-expressions and physiological signals were were annotated and segmented. • Concordance between micro-expressions and physiological signals was investigated. • Significant correlation between micro-expressions and heart rate variability. • Collected data and annotations would be made available for future research. Various modalities have been leveraged for affective computing, alone or combined, such as facial expressions, speech intonations, peripheral physiological signals, and brain activities. Previous studies have shown that the multimodal fusion of affective data usually improves performance. However, the internal interactive mechanism among different modalities is rarely studied. In this paper, we investigated the concordance between facial micro-expressions and physiological signals under high arousal emotion elicitation with strict synchronization. By linking the onset of micro-expressions with physiological signals, a series of epoch durations were created to cover the potential reaction delay that may vary with different physiological signals. The experimental results show a significant correlation between the appearance of micro-expressions and time-domain features of heart rate variability, but not respiration or electrodermal activity related features. These findings indirectly verify the feasibility and reliability of micro-expression as a measure for non-contact genuine emotion recognition and would be beneficial for the fusion of micro-expression and physiological signals for more robust affective computing and their applications in public security and mental health.