Language-emotion interaction modulates selective attention to a speaker’s eyes and mouth: evidence from Chinese-English bilinguals

Abstract

ABSTRACT Speech listeners focus on a speaker's face to acquire different information in social communication. Fixation on the mouth associates with language processing and attention to the eyes is mainly driven by social/emotional cues. Here, we investigated how selective attention to the eyes and mouth would vary with language-emotion interaction during speech processing when unbalanced Chinese-English bilinguals [Chinese, the first language (L1); English, a later-learned language (LX) of upper-intermediate proficiency] viewed a speaker narrating stories of different valence in their L1 and LX. Results showed that when listening to negative stories in Chinese relative to English, bilinguals attended to the eyes more, indicating that LX emotionality is weaker than that in L1, reducing gaze to the eyes where human adults primarily seek for emotional cues. In Chinese, compared with neutral speech, both positive and negative stories increased mouth attention and this effect was not observed in English, suggesting a particularly important role of the mouth in processing emotional information in speech and confirming emotionality reduction in LX. Our study reveals for the first time how linguistic and emotional cues integrated in speech alters attention allocation on speakers’ face to facilitate both speech comprehension and emotional perception in social communication.