Temporal cues from visual information benefit speech perception in noise

Abstract

Speech perception in noise is a challenge for older adults and listeners with hearing loss. It is also challenging for a subset of listeners with normal hearing thresholds. One way to improve performance in noisy settings is to provide visual information of the talker. The current study examines what aspects of this visual input are necessary to receive an audio-visual benefit. To test the hypothesis that a visual analogue of the speech amplitude envelope aids speech comprehension, speech sentences in babble noises, with or without visual stimuli, were presented to listeners. In this case, the visual analogue was a sphere that varies based on normal, mismatched, and reversed relation to the amplitude envelope of the speech signal. Even when the audio-visual correlation is reversed (higher amplitude = smaller volumes), a significant improvement in speech perception in the auditory-visual condition versus the audio-only condition was gained even though no visual representation of phonetic information was available. These results provide strong evidence that the amplitude envelope can be inferred from visual displays and can be integrated online in speech perception. This study has implications for potential technological enhancements to speech perception with hearing devices – in particular, the integration of a non-auditory signal.