The eyes’ footprints on the ears: An investigation of short-term speech intelligibility change.

Abstract

Although observation of mouth movements improves auditory speech perception, the extent to which visual information aids speech perceptual learning and affects subsequent audio-only speech perception remains unknown. The current study investigates whether visual speech information, specifically, the synchronized mouth movements of the talking face during training, helps the listener to perceive audio-only speech more effectively after training. An experimental group was trained to recognize audiovisual words presented in noise, while a control group was trained on the same audio speech signal in noise, but with no accompanying mouth movements. The control group provides a baseline estimate of what listeners can learn from the speech signal alone without visual information from the talker’s mouth movements. Both groups were tested on audio-only speech in noise before and after training. All novel words were used in the pretest, training, and posttest. The results demonstrate that visual information aids learning to recognize audio-only speech in noise, suggesting visual information from a talker’s mouth movements during training might change auditory coding of acoustic speech signals.