Bimodal Speech Research Articles

The effects of signal-to-noise ratio on auditory-visual integration: Integration and encoding are not independent

Most current models of auditory-visual speech perception propose a two-stage process in which unimodal information is extracted independently from each sensory modality and is then combined in a separate integration stage. A central assumption of these models is that integration is a distinct perceptual ability that is separate from the ability to encode unimodal speech information. The purpose of the present study was to evaluate this assumption by measuring integration of the same speech materials across three different signal-to-noise ratios. Twelve participants were presented with 42 repetitions of 13 consonants presented in an /iCi/ environment at 3 different signal-to-noise ratios. Integration was assessed using an optimum processor model [L. Braida, Q. J. Exp. Psych. 43A, 647–677 (1991)] and a new measure termed integration efficiency that is based on a simple probability metric. In contrast to predictions made by current models of auditory-visual speech perception, significant differences were observed for both measures of integration as a function of signal-to-noise ratios. These findings argue against strictly serial models of auditory-visual speech perception and instead support a more interactive architecture in which unimodal encoding interacts with integration abilities to determine overall benefits for bimodal speech perception. [Work supported by NIA.]

Bimodal speech perception in infant hearing aid and cochlear implant users.

To determine the feasibility of replicating prior bimodal perception findings with hearing-impaired infants during their preimplant, hearing aid trial, and postimplant experiences; secondarily, to determine the point in development at which these infants were able to match phonetic information in the lips and voice for the vowels /a/ and /i/. A total of 10 infants with hearing loss, aged 4 to 24 months, were assessed at least once prior to cochlear implantation and previous to implant stimulation. The Split-Screen Preferential Looking Procedure was used to evaluate the bimodal perception skills of these infants. Examples of individual bimodal perception data and preliminary group data are presented. A difference in performance across preimplant and postimplant test sessions was noted for the individuals and the group. These data provide evidence that the infants' audibility levels were improved by their cochlear implants, which may have contributed to their evolving ability to match phonetic information in the lips and voice.

A Fused Hidden Markov Model With Application to Bimodal Speech Processing

This paper presents a novel fused hidden Markov model (fused HMM) for integrating tightly coupled time series, such as audio and visual features of speech. In this model, the time series are first modeled by two conventional HMMs separately. The resulting HMMs are then fused together using a probabilistic fusion model, which is optimal according to the maximum entropy principle and a maximum mutual information criterion. Simulations and bimodal speaker verification experiments show that the proposed model can significantly reduce the recognition errors in noiseless or noisy environments.

Lip geometric features for human–computer interaction using bimodal speech recognition: comparison and analysis

Bimodal speech recognition is a novel extension of acoustic speech recognition for which both acoustic and visual speech information are used to improve the recognition accuracy in noisy environments. Although various bimodal speech systems have been developed, a rigorous and detailed comparison of the possible geometric visual features from speakers' faces has not been given yet in the previous papers. Thus, in this paper, the geometric visual features are compared and analyzed rigorously for their importance in bimodal speech recognition. The relevant information of each possible single visual feature is used to determine the best combination of geometric visual features for both visual-only and bimodal speech recognition. From the geometric visual features analyzed, lip vertical aperture is the most relevant; and the set formed by the vertical and horizontal lip apertures and the first order derivative of the lip corner angle gives the best results among the possibilities of reduced set of geometric features that were analyzed. Also, in this paper, the effect of the modelling parameters of hidden Markov models (HMM) on each single geometric lip feature's recognition accuracy is analyzed. Finally, the accuracy of acoustic-only, visual-only, and bimodal speech recognition methods are experimentally determined and compared using the optimized HMMs and geometric visual features. Compared to acoustic and visual-only speech recognition, the bimodal speech recognition scheme has a much improved recognition accuracy using the geometric visual features, especially in the presence of noise. The results obtained showed that a set of as few as three labial geometric features are sufficient to improve the recognition rate by as much as 20% (from 62%, with acoustic-only information, to 82%, with audio-visual information at a signal to noise ratio (SNR) of 0 dB).

Neural correlates of bimodal speech and gesture comprehension

The present study examined the neural correlates of speech and hand gesture comprehension in a naturalistic context. Fifteen participants watched audiovisual segments of speech and gesture while event-related potentials (ERPs) were recorded to the speech. Gesture influenced the ERPs to the speech. Specifically, there was a right-lateralized N400 effect—reflecting semantic integration—when gestures mismatched versus matched the speech. In addition, early sensory components in bilateral occipital and frontal sites differentiated speech accompanied by matching versus non-matching gestures. These results suggest that hand gestures may be integrated with speech at early and late stages of language processing.

Audiovisual bimodal mutual compensation of Chinese

The perception of human languages is inherently a multi-modal process, in which audio information can be compensated by visual information to improve the recognition performance. Such a phenomenon in English, German, Spanish and so on has been researched, but in Chinese it has not been reported yet. In our experiment, 14 syllables (/ba, bi, bian, biao, bin, de, di, dian, duo, dong, gai, gan, gen, gu/), extracted from Chinese audiovisual bimodal speech database CAVSR-1.0, were pronounced by 10 subjects. The audio-only stimuli, audiovisual stimuli, and visual-only stimuli were recognized by 20 observers. The audio-only stimuli and audiovisual stimuli both were presented under 5 conditions: no noise, SNR 0 dB, −8 dB, −12 dB, and −16dB. The experimental result is studied and the following conclusions for Chinese speech are reached. Human beings can recognize visual-only stimuli rather well. The place of articulation determines the visual distinction. In noisy environment, audio information can remarkably be compensated by visual information and as a result the recognition performance is greatly improved.

Perceptual identification training of American English /r/ and /l/ by Japanese speakers generalizes to novel stimuli and tasks

This study investigated the extent to which results from /r/ and /l/ perceptual identification training generalized to identification of novel natural stimuli and discrimination of synthetic stimuli in ten native Japanese speakers. A behavioral training software program was used which incorporated factors known to affect the acquisition of phonemic distinctions, including bimodal speech cues, stimulus variability, and subject-controlled stimulus presentation with immediate feedback. Natural speech tokens were digitally manipulated to create three levels of acoustic exaggeration. These levels, and other stimulus characteristics such as the number of talkers, vowel contexts, and syllable structure, varied during each training session as a function of listener’s performance. Pre- and post-training identification tasks measured generalization to natural tokens of novel talkers and vowel contexts. In addition, pre- and post-training discrimination tasks measured generalization to novel tasks and synthetic stimuli. Same–different and AXB paradigms were used to measure and compare performance on the trained non-native (/r/–/l/) contrast as well as a native (/b/–/d/) contrast using synthetic CV and VCV tokens previously identified and judged good exemplars by American English listeners. Both identification and discrimination results show generalization of learning to novel stimuli and tasks. [Work supported by NICHD HD37954 to P.K.K.]

Perceptual sensitivity to temporal distortion of visual, auditory and bimodal speech.

Perceptual sensitivity to temporal distortion of visual, auditory and bimodal speech.

Response amplification in sensory-specific cortices during crossmodal binding

Integrating information across the senses can enhance our ability to detect and classify stimuli in the environment. For example, auditory speech perception is substantially improved when the speaker's face is visible. In an fMRI study designed to investigate the neural mechanisms underlying these crossmodal behavioural gains, bimodal (audio-visual) speech was contrasted against both unimodal (auditory and visual) components. Significant response enhancements in auditory (BA 41/42) and visual (V5) cortices were detected during bimodal stimulation. This effect was found to be specific to semantically congruent crossmodal inputs. These data suggest that the perceptual improvements effected by synthesizing matched multisensory inputs are realised by reciprocal amplification of the signal intensity in participating unimodal cortices.

Bimodal Speech Perception by Native and Nonnative Speakers of English: Factors Influencing the McGurk Effect

This research provides a multidisciplinary perspective on the factors influencing the process of integrating auditory and visual information in speech perception, and the nature of the mental representations to which speech input is matched for identification. Most previous studies of L2 speech perception had focused on only one source of input—the auditory modality. In this study, however, the occurrence of the McGurk effect (the influence of visual or lip‐read information on speech perception) pointed to the power of visible speech in face‐to‐face communication for L2 learners as well as native speakers. Participants were advanced ESL learners of 4 L1s (Japanese, Korean, Spanish, and Malay) and native speakers of English. In this experimental design, six CV syllables with /p, f, w, r, t, k/ were combined on videotape to create concordant and discordant conditions (i.e., where audiovisual cues were matched and mismatched). This design permits the tagging of input according to its source in order to evaluate the relative contributions of each modality to the perceptual outcome. Results indicated that the learners' L1s influenced the relative information value of the auditory and visual cues, and thus their contribution to a single percept. The Japanese and Korean learners' identification accuracy of /f/ and /r/ increased with matched visual cues. It was evident that learners attended to visual cues to identify speech sounds even though they had not received any explicit training.By drawing from the literature on L2 phonology, speechreading, speech perception, attention, similarity, and categorization, this work demonstrates the need for a model of L2 speech acquisition within a cognitive framework to account for bimodal input in the broader context of spoken language processing. Findings also point to the need to incorporate into perceptual training for L2 learners the enhancement of the information value of visual cues as a second channel of input.

Complementarity and synergy in bimodal speech: Auditory, visual, and audio-visual identification of French oral vowels in noise

The efficacy of audio-visual interactions in speech perception comes from two kinds of factors. First, at the information level, there is some "complementarity" of audition and vision: It seems that some speech features, mainly concerned with manner of articulation, are best transmitted by the audio channel, while some other features, mostly describing place of articulation, are best transmitted by the video channel. Second, at the information processing level, there is some "synergy" between audition and vision: The audio-visual global identification scores in a number of different tasks involving acoustic noise are generally greater than both the auditory-alone and the visual-alone scores. However, these two properties have been generally demonstrated until now in rather global terms. In the present work, audio-visual interactions at the feature level are studied for French oral vowels which contrast three series, namely front unrounded, front rounded, and back rounded vowels. A set of experiments on the auditory, visual, and audio-visual identification of vowels embedded in various amounts of noise demonstrate that complementarity and synergy in bimodal speech appear to hold for a bundle of individual phonetic features describing place contrasts in oral vowels. At the information level (complementarity), in the audio channel the height feature is the most robust, backness the second most robust one, and rounding the least, while in the video channel rounding is better than height, and backness is almost invisible. At the information processing (synergy) level, transmitted information scores show that all individual features are better transmitted with the ear and the eye together than with each sensor individually.

Bimodal Speech Perception by Native and Nonnative Speakers of English: Factors Influencing the McGurk Effect

Two experiments explored factors affecting the influence of visual (lip‐read) information on auditory speech perception, the “McGurk effect”, in 120 advanced ESL learners of 4 L1s (Japanese, Korean, Spanish, and Malay) and 50 native speakers (NSs) of English. The audio and video speech signals of a female English speaker producing CV syllables with /p,f,w,r,t,k/ and// were combined on videotape. For nonnative speakers (NNSs), identification accuracy of /f/ and /r/ increased with matched visual cues. Visual nonlabials /t,k/ significantly influenced perception of auditory labials. For NSs, significant effects were noted only in noise. When stimuli involved only /p,t,k/, NSs and NNSs reported significant visual effects. Results demonstrate the influence of first language (L1) and linguistic experience on the relative information value of the cues, assumption of perceptual unity, and thus audiovisual integration. Implications for bimodal speech perception and second language instruction are discussed.

Perceiving speech from inverted faces.

We examined whether the orientation of the face influences speech perception in face-to-face communication. Participants identified auditory syllables, visible syllables, and bimodal syllables presented in an expanded factorial design. The syllables were /ba/, /va/, /oa/, or /da/. The auditory syllables were taken from natural speech whereas the visible syllables were produced by computer animation of a realistic talking face. The animated face was presented either as viewed in normal upright orientation or inverted orientation (180 degrees frontal rotation). The central intent of the study was to determine if an inverted view of the face would change the nature of processing bimodal speech or simply influence the information available in visible speech. The results with both the upright and inverted face views were adequately described by the fuzzy logical model of perception (FLMP). The observed differences in the FLMP's parameter values corresponding to the visual information indicate that inverting the view of the face influences the amount of visible information but does not change the nature of the information processing in bimodal speech perception.

Cross-linguistic comparisons in the integration of visual and auditory speech

We examined how speakers of different languages perceive speech in face-to-face communication. These speakers identified synthetic unimodal and bimodal speech syllables made from synthetic auditory and visual five-step /ba/-/da/ continua. In the first experiment, Dutch speakers identified the test syllables as either /ba/ or /da/. To explore the robustness of the results, Dutch and English speakers were given a completely open-ended response task. Tasks in previous studies had always specified a set of alternatives. Similar results were found in the two-alternative and open-ended task. Identification of the speech segments was influenced by both the auditory and the visual sources of information. The results falsified an auditory dominance model (ADM) which assumes that the contribution of visible speech is dependent on poor-quality audible speech. The results also falsified an additive model of perception (AMP) in which the auditory and visual sources are linearly combined. The fuzzy logical model of perception (FLMP) provided a good description of performance, supporting the claim that multiple sources of continuous information are evaluated and integrated in speech perception. These results replicate previous results found with English, Spanish, and Japanese speakers. Although there were significant performance differences, the model analyses indicated no differences in the nature of information processing across language groups. The performance differences across languages were caused by information differences due to different phonologies in Dutch and English. These results suggest that the underlying mechanisms for speech perception are similar across languages.

Bimodal speech perception: an examination across languages

We examined whether language and culture influence speech perception in face-to-face communication. Native speakers of Japanese, Spanish and English identified the same synthetic unimodal and bimodal speech syllables. Five-step /ba/–/da/ continua were synthesized along auditory and visual dimensions, by varying properties of the syllable at its onset. In the first experiment, the three language groups identified the test syllables as /ba/ or /da/; in the second, Japanese and English speakers were given an open-ended set of response alternatives. For all language groups, identification of the speech segments was influenced by both auditory and visual sources of information. Given the results, we were able to reject an auditory dominance model (ADM) which assumes that the contribution of visible speech is dependent on poor-quality audible speech. The results also falsified a categorical model of perception (CMP) in which the auditory and visual sources are categorized before they are combined. The fuzzy logical model of perception (FLMP) provided a good description of performance supporting the claim that multiple sources of continuous information are evaluated and integrated in speech perception. No differences in the nature of processing across language groups suggests that the underlying mechanisms for speech perception are similar across language and culture.

Perceiving asynchronous bimodal speech in consonant-vowel and vowel syllables

Subjects naturally integrate auditory and visual information in bimodal speech perception. To assess the robustness of the integration process, the relative onset time of the audible and visible sources was systematically varied. In the first experiment, bimodal syllables composed of the auditory and visible syllables /ba/ and /da/ were present at five different onset asynchronies. The second experiment replicated the same procedure but with the vowels /i/ and /u/. The results indicated that perceivers integrated the two sources of information at all asynchronies. Cluster responses (for example, /bda/ given visual /ba/ and auditory /da/) occurred primarily for the consonants but not for the vowels. In addition, cluster responses require that both the visual and the auditory information be reasonable compatible with the physical properties of a cluster articulation. For both vowels and consonant-vowel syllables, information from the auditory and visual sources is continuous, independent and combined in a three-stage process of feature evaluation, integration and decision.

Long-term training, transfer, and retention in learning to lipread.

A long-term training paradigm in lipreading was used to test the fuzzy logical model of perception (FLMP). This model has been used successfully to describe the joint contribution of audible and visible speech in bimodal speech perception. Tests of the model were extended in the present experiment to include the prediction of confusion matrices, as well as performance at several different levels of skill. The predictions of the FLMP were contrasted with the predictions of a prelabeling integration model (PRLM). Subjects were taught to lipread 22 initial consonants in three different vowel contexts. Training involved a variety of discrimination and identification lessons with the consonant-vowel syllables. Repeated testing was given on syllables, words, and sentences. The test items were presented visually, auditorily, and bimodally, at normal rate or three times normal rate. The subjects improved in their lipreading ability across all three types of test items. Replicating previous results, the present study illustrates that substantial gains in lipreading performance are possible. Relative to the PRLM, the FLMP gave a better description of the confusion matrices at both the beginning and the end of practice. One new finding from the present study is that the FLMP can account for the gains in bimodal speech perception as subjects improve their lipreading and listening abilities.

La langue et les Lèvres: cross-language influences on bimodal speech perception.

Previous research in speech perception has yielded two sets of findings which are brought together in the present study. First, it has been shown that normal hearing listeners use visible as well as acoustical information when processing speech. Second, it has been shown that there is an effect of specific language experience on speech perception such that adults often have difficulty identifying and discriminating non-native phones. The present investigation was designed to extend and combine these two sets of findings. Two studies were conducted using six consonant-vowel syllables (/ba/, /va/, /alpha a/, /da/, /3a/, and /ga/ five of which occur in French and English, and one (the interdental fricative /alpha a/) which occurs only in English. In Experiment 1, an effect of specific linguistic experience was evident for the auditory identification of the non-native interdental stimulus by French-speakers. In Experiment 2, it was shown that the effect of specific language experience extends to the perception of the visible information in speech. These findings are discussed in terms of their implications for our understanding of cross-language processes in speech perception and for our understanding of the development of bimodal speech perception.

Developmental changes in visual and auditory contributions to speech perception

Previous research has demonstrated that listeners make use of both auditory and visual information in bimodal speech perception. Preschool children appear to evaluate and integrate the two sources of information in the same manner as adults, but the children are less influenced by the visual source. The current experiments test a possible explantation of this difference and extend the question to younger children. It is possible that children are poorer lip-readers than adults and thus have less complete information about the visual source. Children and adults were tested with both auditory and visual sources and were also required to identify speech events on the basis of only the visual source. In addition to replicating the previous findings in the bimodal situation, the current experiments found that children are poorer lip-readers than adults. Furthermore, a positive correlation was observed between lip-reading ability and the size of the visual contribution to bimodal speech perception. A fuzzy logical model of speech perception provided a good quantitative description of the results even with the assumption that the visual information was equivalent in both the bimodal and lip-reading conditions. The results also contradict the categorical perception of speech events and any nonindependence in the evaluation of auditory and visual information in speech perception.

Bimodal speech perception in early infancy

We previously reported preliminary data on infants' abilities to detect the cross‐modal correspondences between the visual and auditory concomittants of speech [Kuhl and Meltzoff, J. Acoust. Soc. Am. Suppl. 1 70, S96 (1981)]. We have now completed testing on 32 4.5‐ to 5‐month‐old infants. Each was shown a filmed display of two faces, one producing the articulatory movements corresponding to the vowel [a] and the other producing the articulatory movements corresponding to the vowel [i]. One of the sound tracks (either [a] or [i]) was played in synchrony with the faces. The infants' visual fixations to the faces were scored by an observer who could not hear the sound track presented to the infant nor see the faces. Results demonstrated that the infants looked longer at the face matching the sound being presented than at the nonmatching face (p < 0.01). We hypothesized that the recognition of these cross‐modal equivalences was based on the structural correspondence between a particular articulatory movement and a particular vowel sound, rather than on any temporal correspondence between a particular face‐sound pair. This hypothesis predicts that if the crucial spectral information is removed from the vowels while the temporal information is preserved, performance should drop to chance. Experiment II tested this hypothesis using pure‐tone stimuli. Performance fell to chance. [Supported by NSF.]