Audiovisual integration in familiar person recognition

Abstract

ABSTRACTInterpersonal communication is a multimodal experience. Perceptual integration of information conveyed by voice and face has previously been demonstrated for speech comprehension and speaker localization. For instance, audiovisual (McGurk) illusions occur when perceiving an auditory syllable with a time-synchronized incongruent syllable in a visual facial articulation. However, despite a large body of research on audiovisual integration (AVI), researchers have only recently begun to consider the role of AVI for person recognition. In the present paper, Experiment 1 demonstrated systematic benefits and costs for recognition of familiar voices when combined with time-synchronized articulating faces of corresponding or noncorresponding speaker identity, respectively. By contrast, no costs (relative to auditory-only presentation) occurred when voices were combined with static noncorresponding faces. Moreover, this pattern was clear for familiar but not unfamiliar voices. In Experiment 2, smaller benefits and costs for familiar voices when combined with time-reversed (as compared to time-synchronized) facial articulations suggest a role of synchrony for AVI, over and above effects of facial motion per se. Experiment 3 demonstrated benefits, as well as a trend for costs, of familiar face recognition when faces were presented with voices of corresponding or noncorresponding identity. Overall, these results suggest that AVI is an important factor in the recognition of people, depends on familiarity with a speaker, shows sensitivity to temporal synchronization of the facial and vocal articulation, and can occur in a bidirectional manner.