The effects of signal-to-noise ratio on auditory-visual integration: Integration and encoding are not independent

Abstract

Most current models of auditory-visual speech perception propose a two-stage process in which unimodal information is extracted independently from each sensory modality and is then combined in a separate integration stage. A central assumption of these models is that integration is a distinct perceptual ability that is separate from the ability to encode unimodal speech information. The purpose of the present study was to evaluate this assumption by measuring integration of the same speech materials across three different signal-to-noise ratios. Twelve participants were presented with 42 repetitions of 13 consonants presented in an /iCi/ environment at 3 different signal-to-noise ratios. Integration was assessed using an optimum processor model [L. Braida, Q. J. Exp. Psych. 43A, 647–677 (1991)] and a new measure termed integration efficiency that is based on a simple probability metric. In contrast to predictions made by current models of auditory-visual speech perception, significant differences were observed for both measures of integration as a function of signal-to-noise ratios. These findings argue against strictly serial models of auditory-visual speech perception and instead support a more interactive architecture in which unimodal encoding interacts with integration abilities to determine overall benefits for bimodal speech perception. [Work supported by NIA.]