Modeling the interaction of phonemic intelligibility and lexical structure in audiovisual word recognition

Abstract

Studies of audiovisual perception of spoken language have mostly modeled phoneme identification in nonsense syllables, but it is doubtful that models or theories of phonetic processing can adequately account for audiovisual word recognition. The present study took a computational approach to examine how lexical structure may additionally constrain word recognition, given the phonetic information available under vocoded audio, visual and audiovisual stimulus conditions. Subjects made phonemic identification judgments on recordings of spoken nonsense syllables. Hierarchical cluster analysis was used first to select classes of perceptually equivalent phonemes for each of the stimulus conditions, and then a machine-readable phonemically transcribed lexicon was retranscribed in terms of these phonemic equivalence classes. Several statistics were computed for each of the transcriptions, including percent information extracted, percent words unique and expected class size. The findings suggest that superadditive levels of audiovisual enhancement are more likely for monosyllabic than for multisyllabic words. That is, impoverished phonetic information may be sufficient to recognize multisyllabic words, but the recognition of monosyllabic words seems to require additional phonetic information.