Multimodal recognition of interrupted speech: Benefit from text and visual speech cues

Abstract

Presenting degraded speech with visual cues facilitates speech recognition. This benefit is observed for visual speech cues that are perceptually correlated with the auditory signal, as well as for text cues that delay integration until a later cognitive-linguistic processing stage. However, it is not clear how the benefit compares between these two types of degraded multimodal presentations. The current study examined how listeners integrate visually interrupted text or visual speech cues with acoustically interrupted speech. In Experiment 1, text was periodically interrupted by white space at visual interruption rates that were associated with the auditory interruption rate of speech. In Experiment 2, videos were visually interrupted by grey frames. The synchrony of audio-visual interruption was also manipulated by presenting visual cues in-phase or 180o out-of-phase with speech interruptions. For both experiments, speech was low-pass filtered at 2000 Hz. Preliminary results indicate that listeners obtain a benefit from both visual speech and text cues. In addition, performance is affected by the interruption rate of speech, with minimal performance obtained around an interruption rate of 2 Hz. Supplementing speech with incomplete visual cues can improve sentence intelligibility and compensate for degraded speech in adverse listening conditions. [Work supported, in part, by NIH/NIDCD.]