Detection and resolution of audio‐visual conflict in the perception of vowels

Abstract

To determine whether optical cues can influence the identification of acoustically defined vowels, three series of /bVd/ syllables were synthesized with the medial vowel ranging from /u/ to /a/, /a/ to /i/, and /i/ back to /u/. The members of each series were synchronized with video‐recordings of the face of a (natural) talker uttering the end‐point syllables of that series, and were presented to adults with normal hearing and vision (a) to obtain ratings of audio‐visual compatibility, and (b) for forced choice identification. Perceived compatibility decreased essentially monotonically across each series as the acoustical syllable became less like that normally appropriate for the syllable displayed optically. Nonetheless, systematic changes in patterns of identification occurred between audio‐visual and audio‐alone presentations, with audio‐visual vowels identified as more like the vowel presented visually. These results demonstrate that analogous effects obtained previously with consonants are not restricted to a particular phonetic class, and are compatible with theoretical arguments [Summerfield, Phonetica 36, 314–331 (1979)] that perceptual integration of information in the two modalities occurs prior to phonetic categorization.