Modality salience in multimodal speech perception under degraded presentation conditions

Abstract

Studies of the McGurk effect in auditory-visual speech perception have demonstrated that both modalities contribute to the overall percept, even when both modalities are perfectly rendered. Initial studies in our laboratory found that a McGurk-type effect can also be observed with tactile-visual presentation, employing a tactile speech perception device. The present study investigated the question of whether subjects’ relative reliance on a particular sensory modality could be shifted when presentation via one modality was degraded. Subjects were tested with a McGurk paradigm in unimodal (visual, auditory, or tactile) and multimodal presentations, in both nondegraded and degraded presentation conditions. Auditory degradation was achieved by low-pass filtering with cutoffs at 400 or 1000 Hz, and visual degradation was achieved by use of a diffusing screen placed in front of a video monitor. Tactile stimuli were not degraded. Results showed that subjects quickly shifted response patterns when one modality was degraded to show increased reliance on the nondegraded modality. However, increased reliance on the nondegraded tactile input was not found, possibly because the initial training with the tactile device was limited. Nonetheless, tactile-visual McGurk effects were replicated. [Portions of this work were supported by NIH.]