Predictive power in models of audiovisual integration of speech

Abstract

Seeing the talking face can influence the phoneme perceived from the voice. This facilitates speech perception in the natural case where the face and voice are congruent and can cause the McGurk illusion when they are not. The classical example of the McGurk illusion is when acoustic /aba/ is perceived as /ada/ when dubbed onto a face articulating /aga/. In order to fully understand the underlying process of integrating information across the senses we need a computational account with predictive power. The Fuzzy Logical Model of Perception is one computational account of audiovisual integration in speech perception. Here we describe alternative accounts in which integration is based on an early continuous internal representation on which the phonetic classes fall. We show that these alternative accounts can provide just as good a fit when corrected for the number of free parameters. We also show, using cross-validation, that they have greater, but not great, predictive power. Finally, we show that introducing a regularization term can amend the lack of predictive power. With regularization, models based on continuous representations have the highest predictive power.