Cross-modal integration in speech and nonspeech tasks

Abstract

Studies of audiovisual speech perception show that both modalities contribute to the speech perception process. One salient example of this is the McGurk effect, in which visual /ga/ and auditory /ba/ might produce the fused percept of /da/. The present study investigated whether similar multimodal integration occurs for tactile–visual presentation via tactile speech perception devices, and whether these integration effects are specific to speech, or can also occur for nonspeech stimuli. The first experiment tested visual, auditory, tactile, and multimodal presentation of speech and nonspeech stimuli. Results indicated McGurk-type integration effects for both audiovisual and tactile–visual speech tokens, as well as some evidence of similar effects for nonspeech stimuli. Overall levels of tactile–visual integration were lower than audiovisual levels, possibly because of minimal subject training with the tactile aid. A second experiment provided two months of tactile aid training prior to integration testing. Preliminary results show an increase in McGurk-type integration for tactile–visual presentation. Overall findings suggest that the integration process is not restricted to audiovisual presentation. [Work supported by NIH.]