Investigating the audio–visual speech detection advantage

Abstract

Seeing the moving face of the talker permits better detection of speech in noise compared to auditory only presentation, an Audio–Visual (AV) facilitation effect. Three experiments that used a masked speech detection task are reported. The experiments were designed to contrast two accounts of the AV facilitation effect (AV peak listening and AV grouping). In each experiment a different manipulation of the relationship between the auditory and visual signals was employed. The first experiment manipulated the sequencing of the visual and auditory information by presenting the displays time normal or time reversed. The results showed that AV facilitation only occurred for the time normal presentations where there was a high correlation between the AV signals. Experiment 2 examined the impact on AV facilitation of shifting the auditory signals earlier in time than its normal position (again with time normal and time reversed presentation). It was found that shifting the auditory component abolished the AV effect for both the time normal and reversed conditions. The final experiment examined the AV detection advantage using another situation in which the relationship between the AV signals differed. Two versions of AV speech produced by a virtual talker were investigated. In one version, based on text-to-speech synthesis, the video and auditory signals were more rapid than the human talker of Experiment 1. In the other version, the signals were lengthened to match the durations of the human talker. A small but reliable AV facilitation effect was only found for the second version. The results are consistent with a cross-modal peak listening account and are discussed in terms of constraints on the integration of auditory and visual speech.