The effects of audiovisual speech on sentential phoneme detection

Abstract

Many previous studies in audiovisual speech perception concentrate on situations where auditory information is degraded, or stimuli with conflicting auditory and visual information. However, relatively little work has addressed the issue of how the availability of visual information contributes to postphonetic levels of processing. Presented here are results from a series of studies targeting audiovisual processing in lexical and sentential structures. Subjects in a phoneme-monitoring task for target words in sentences showed significantly faster RTs for audiovisual conditions as compared to auditory-only conditions. Additionally, a lexical frequency effect was observed in the auditory-only, but not the audiovisual condition. These initial findings focused on the bilabial viseme group and were compatible with an interpretation based on the earlier temporal availability of the visual information as compared to the auditory. Subsequent experimentation with the same general procedures incorporated catch trials with nontarget bilabials, and indicates a more complex account. Comparison of the original findings with other viseme groups also supports the view that the availability of visual information contributes more than a simple temporally earlier trigger for target identification. Monitoring performance cannot be well accounted for by treating the auditory and visual information as separate channels for controlling the response.