Abstract
Audiovisual perception of emotions has been typically examined using displays of a solitary character (e.g., the face-voice and/or body-sound of one actor). However, in real life humans often face more complex multisensory social situations, involving more than one person. Here we ask if the audiovisual facilitation in emotion recognition previously found in simpler social situations extends to more complex and ecological situations. Stimuli consisting of the biological motion and voice of two interacting agents were used in two experiments. In Experiment 1, participants were presented with visual, auditory, auditory filtered/noisy, and audiovisual congruent and incongruent clips. We asked participants to judge whether the two agents were interacting happily or angrily. In Experiment 2, another group of participants repeated the same task, as in Experiment 1, while trying to ignore either the visual or the auditory information. The findings from both experiments indicate that when the reliability of the auditory cue was decreased participants weighted more the visual cue in their emotional judgments. This in turn translated in increased emotion recognition accuracy for the multisensory condition. Our findings thus point to a common mechanism of multisensory integration of emotional signals irrespective of social stimulus complexity.
Highlights
Perception of emotions is a multimodal event; by integrating signals from facial expressions, body movements, vocal prosody and other cues, we make emotional judgments about others. This multisensory integration of emotional expressions has been studied with faces and voices, body expression and faces (Meeren et al, 2005; Van den Stock et al, 2007), body expression with sound stimuli (Vines et al, 2006; Petrini et al, 2010, 2011), and body expressions and voices (Pichon et al, 2008; Stienen et al, 2011)
We found no significant effect of factor “attention” [F(1, 15) = 0.11, p = 0.74, ηG2 = 0] indicating that the level of accuracy for emotion recognition did not depend on the specific modality attended
In the present study we ask whether the multisensory facilitation in emotion recognition, reported by previous studies using single agent social displays, extends to multiagent social interactions
Summary
Perception of emotions is a multimodal event; by integrating signals from facial expressions, body movements, vocal prosody and other cues, we make emotional judgments about others This multisensory integration of emotional expressions has been studied with faces and voices (de Gelder and Vroomen, 2000; Kreifelts et al, 2007; Collignon et al, 2008), body expression and faces (Meeren et al, 2005; Van den Stock et al, 2007), body expression with sound stimuli (Vines et al, 2006; Petrini et al, 2010, 2011), and body expressions and voices (Pichon et al, 2008; Stienen et al, 2011). Collignon et al (2008) showed that participants were faster and more accurate to identify fearful and disgust expressions when they observed faces combined with voices than either faces or voices alone This multisensory behavioral facilitation became evident when the most reliable visual information was degraded, changing the participants weighting strategy (i.e., they weighted the auditory cue more when judging the expressed emotion). A small number of studies have examined how observers integrate signals from emotional body movement and voice, and results so far follow a similar pattern to studies of emotional faces and voices (Van den Stock et al, 2007)
Sign-in/Register to view highlights & summary 
Published Version (
Free)
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call