Integration of Facial and Vocal Cues in the Perception of Bimodal Emotion Changes

Abstract

How emotion changes in vocal and facial modalities integrate into a unified percept has yet to be investigated. We tested this question by recording electroencephalogram (EEG) while participants detect vocal or facial emotion changes delivered by facial, vocal, and facial–vocal expressions. The detection of emotion change, regardless of modalities, was associated with larger N2 and P3, as well as greater theta synchronization and beta desynchronization. Importantly, we observed significant superadditive responses in P3 amplitudes and theta oscillatory power during the detection of bimodal emotional changes and such superadditive responses were modulated by task focus. These results demonstrated a robust effect of facial–vocal bimodal integration during emotional change detection, which is most likely modulated by task focus.