Auditory-visual Speech Perception Research Articles

Speech perception: Auditory and visual cue integration in children with and without phonological disorder in voiceless fricatives

ABSTRACT The literature reports contradictory results regarding the influence of visual cues on speech perception tasks in children with phonological disorder (PD). This study aimed to compare the performance of children with (n = 15) and without PD (n = 15) in audiovisual perception task in voiceless fricatives. Assuming that PD could be associated with an inability to integrate phonological information from two sensory sources, we presumed that children with PD would present difficulties in integrating auditory and visual cues compared to typical children. A syllable identification task was conducted. The stimuli were presented according to four conditions: auditory-only (AO); visual-only (VO); audiovisual congruent (AV+); and audiovisual incongruent (AV-). The percentages of correct answers and the respective reaction times in the AO, VO, and AV+ conditions were considered for the analysis. The correct percentage of auditory stimuli was considered for the AV- condition, as well as the percentage of perceptual preference: auditory, visual, and/or illusion (McGurk effect), with the respective reaction time. In comparing the four conditions, children with PD presented a lower number of correct answers and longer reaction time than children with typical development, mainly for the VO. Both groups showed a preference for auditory stimuli for the AV- condition. However, children with PD showed higher percentages for visual perceptual preference and the McGurk effect than typical children. The superiority of typical children over PD children in auditory-visual speech perception depends on type of stimuli and condition of presentation.

How do human newborns come to understand the multimodal environment?

For a long time, newborns were considered as human beings devoid of perceptual abilities who had to learn with effort everything about their physical and social environment. Extensive empirical evidence gathered in the last decades has systematically invalidated this notion. Despite the relatively immature state of their sensory modalities, newborns have perceptions that are acquired, and are triggered by, their contact with the environment. More recently, the study of the fetal origins of the sensory modes has revealed that in utero all the senses prepare to operate, except for the vision mode, which is only functional starting from the first minutes after birth. This discrepancy between the maturation of the different senses leads to the question of how human newborns come to understand our multimodal and complex environment. More precisely, how the visual mode interacts with the tactile and auditory modes from birth. After having defined the tools that newborns use to interact with other sensory modalities, we review studies across different fields of research such as the intermodal transfer between touch and vision, auditory-visual speech perception, and the existence of links between the dimensions of space, time, and number. Overall, evidence from these studies supports the idea that human newborns are spontaneously driven, and cognitively equipped, to link information collected by the different sensory modes in order to create a representation of a stable world.

Multisensory benefits for speech recognition in noisy environments.

A series of our previous studies explored the use of an abstract visual representation of the amplitude envelope cues from target sentences to benefit speech perception in complex listening environments. The purpose of this study was to expand this auditory-visual speech perception to the tactile domain. Twenty adults participated in speech recognition measurements in four different sensory modalities (AO, auditory-only; AV, auditory-visual; AT, auditory-tactile; AVT, auditory-visual-tactile). The target sentences were fixed at 65 dB sound pressure level and embedded within a simultaneous speech-shaped noise masker of varying degrees of signal-to-noise ratios (-7, -5, -3, -1, and 1 dB SNR). The amplitudes of both abstract visual and vibrotactile stimuli were temporally synchronized with the target speech envelope for comparison. Average results showed that adding temporally-synchronized multimodal cues to the auditory signal did provide significant improvements in word recognition performance across all three multimodal stimulus conditions (AV, AT, and AVT), especially at the lower SNR levels of -7, -5, and -3 dB for both male (8-20% improvement) and female (5-25% improvement) talkers. The greatest improvement in word recognition performance (15-19% improvement for males and 14-25% improvement for females) was observed when both visual and tactile cues were integrated (AVT). Another interesting finding in this study is that temporally synchronized abstract visual and vibrotactile stimuli additively stack in their influence on speech recognition performance. Our findings suggest that a multisensory integration process in speech perception requires salient temporal cues to enhance speech recognition ability in noisy environments.

Age Differences in the Effects of Speaking Rate on Auditory, Visual, and Auditory-Visual Speech Perception.

This study was designed to examine how speaking rate affects auditory-only, visual-only, and auditory-visual speech perception across the adult lifespan. In addition, the study examined the extent to which unimodal (auditory-only and visual-only) performance predicts auditory-visual performance across a range of speaking rates. The authors hypothesized significant Age × Rate interactions in all three modalities and that unimodal performance would account for a majority of the variance in auditory-visual speech perception for speaking rates that are both slower and faster than normal. Participants (N = 145), ranging in age from 22 to 92, were tested in conditions with auditory-only, visual-only, and auditory-visual presentations using a closed-set speech perception test. Five different speaking rates were presented in each modality: an unmodified (normal rate), two rates that were slower than normal, and two rates that were faster than normal. Signal to noise ratios were set individually to produce approximately 30% correct identification in the auditory-only condition and this signal to noise ratio was used in the auditory-only and auditory-visual conditions. Age × Rate interactions were observed for the fastest speaking rates in both the visual-only and auditory-visual conditions. Unimodal performance accounted for at least 60% of the variance in auditory-visual performance for all five speaking rates. The findings demonstrate that the disproportionate difficulty that older adults have with rapid speech for auditory-only presentations can also be observed with visual-only and auditory-visual presentations. Taken together, the present analyses of age and individual differences indicate a generalized age-related decline in the ability to understand speech produced at fast speaking rates. The finding that auditory-visual speech performance was almost entirely predicted by unimodal performance across all five speaking rates has important clinical implications for auditory-visual speech perception and the ability of older adults to use visual speech information to compensate for age-related hearing loss.

Assessing perceived listening difficulty using behavioral gaze patterns for audiovisual speech

Listeners alternate their gaze between the talker’s eyes, nose, and mouth during conversation, spending more time gazing directly at the mouth in more difficult listening conditions. Perceived listening difficulty and speech perception performance vary independently, demonstrated with indirect measures of perceived listening difficulty/effort such as pupil dilation and dual task paradigms. This study took a more direct approach, evaluating whether perceived listening difficulty could be assessed by observing a listener’s natural gaze pattern in a challenging speech-perception task. The proportion of time spent gazing at the talker’s mouth was measured for auditory-visual speech perception in conditions that produced equal percentage-correct scores, but where one condition was designed to produce greater perceived difficulty. Experiment 1 compared gaze proportion for a speech-shaped noise versus two same-gender interfering talkers. Experiment 2 examined the effect of sentence context using low and high-probability sentences. The results are discussed in terms of the potential application of the eye-gaze metric to evaluate rehabilitative interventions in cases where subjectively reported benefits do not manifest in improved speech-understanding scores. [The views expressed in this abstract are those of the authors and do not reflect the official policy of the Department of Army/Navy/Air Force, Department of Defense, or U.S. Government.]

Language-General Auditory-Visual Speech Perception: Thai-English and Japanese-English McGurk Effects.

Cross-language McGurk Effects are used to investigate the locus of auditory-visual speech integration. Experiment 1 uses the fact that [], as in 'sing', is phonotactically legal in word-final position in English and Thai, but in word-initial position only in Thai. English and Thai language participants were tested for 'n' perception from auditory [m]/visual [] (A[m]V[]) in word-initial and -final positions. Despite English speakers' native language bias to label word-initial [] as 'n', the incidence of 'n' percepts to A[m]V[] was equivalent for English and Thai speakers in final and initial positions. Experiment 2 used the facts that (i) [ð] as in 'that' is not present in Japanese, and (ii) English speakers respond more often with 'tha' than 'da' to A[ba]V[ga], but more often with 'di' than 'thi' to A[bi]V[gi]. English and three groups of Japanese language participants (Beginner, Intermediate, Advanced English knowledge) were presented with A[ba]V[ga] and A[bi]V[gi] by an English (Experiment 2a) or a Japanese (Experiment 2b) speaker. Despite Japanese participants' native language bias to perceive 'd' more often than 'th', the four groups showed a similar phonetic level effect of [a]/[i] vowel context× 'th' vs. 'd' responses to A[b]V[g] presentations. In Experiment 2b this phonetic level interaction held, but was more one-sided as very few 'th' responses were evident, even in Australian English participants. Results are discussed in terms of a phonetic plus postcategorical model, in which incoming auditory and visual information is integrated at a phonetic level, after which there are post-categorical phonemic influences.

Auditory-visual speech perception in three- and four-year-olds and its relationship to perceptual attunement and receptive vocabulary.

Despite the body of research on auditory-visual speech perception in infants and schoolchildren, development in the early childhood period remains relatively uncharted. In this study, English-speaking children between three and four years of age were investigated for: (i) the development of visual speech perception - lip-reading and visual influence in auditory-visual integration; (ii) the development of auditory speech perception and native language perceptual attunement; and (iii) the relationship between these and a language skill relevant at this age, receptive vocabulary. Visual speech perception skills improved even over this relatively short time period. However, regression analyses revealed that vocabulary was predicted by auditory-only speech perception, and native language attunement, but not by visual speech perception ability. The results suggest that, in contrast to infants and schoolchildren, in three- to four-year-olds the relationship between speech perception and language ability is based on auditory and not visual or auditory-visual speech perception ability. Adding these results to existing findings allows elaboration of a more complete account of the developmental course of auditory-visual speech perception.

Comparative study of difference in auditory-visual speech perception between Japanese and Korean

Comparative study of difference in auditory-visual speech perception between Japanese and Korean

Enhanced audiovisual integration with aging in speech perception: a heightened McGurk effect in older adults

Two experiments compared young and older adults in order to examine whether aging leads to a larger dependence on visual articulatory movements in auditory-visual speech perception. These experiments examined accuracy and response time in syllable identification for auditory-visual (AV) congruent and incongruent stimuli. There were also auditory-only (AO) and visual-only (VO) presentation modes. Data were analyzed only for participants with normal hearing. It was found that the older adults were more strongly influenced by visual speech than the younger ones for acoustically identical signal-to-noise ratios (SNRs) of auditory speech (Experiment 1). This was also confirmed when the SNRs of auditory speech were calibrated for the equivalent AO accuracy between the two age groups (Experiment 2). There were no aging-related differences in VO lipreading accuracy. Combined with response time data, this enhanced visual influence for the older adults was likely to be associated with an aging-related delay in auditory processing.

The relationship between auditory–visual speech perception and language-specific speech perception at the onset of reading instruction in English-speaking children

Speech perception is auditory–visual, but relatively little is known about auditory–visual compared with auditory-only speech perception. One avenue for further understanding is via developmental studies. In a recent study, Sekiyama and Burnham (2008) found that English speakers significantly increase their use of visual speech information between 6 and 8 years of age but that this development does not appear to be universal across languages. Here, the possible bases for this language-specific increase among English speakers were investigated. Four groups of English-language children (5, 6, 7, and 8years) and a group of adults were tested on auditory–visual, auditory-only, and visual-only speech perception; language-specific speech perception with native and non-native speech sounds; articulation; and reading. Results showed that language-specific speech perception and lip-reading ability reliably predicted auditory–visual speech perception in children but that adult auditory–visual speech perception was predicted by auditory-only speech perception. The implications are discussed in terms of both auditory–visual speech perception and language development.

Visual benefit in bimodal training with highly distorted speech sound

Artificial auditory devices such as cochlear implants (CIs) and auditory brainstem implants (ABIs) have become standard means to manage profound sensorineural hearing loss. However, because of their structural limitations compared to the cochlea and the cochlear nucleus, the generated auditory sensations are still imperfect. Recipients need postoperative auditory rehabilitation. To improve these rehabilitation programs, this study evaluated the effects of bimodal (audio–visual) training under seven experimental conditions of distorted speech sound, named noise-vocoded speech sound (NVSS), which is similarly processed with a speech processor of CI/ABI. Word intelligibilities under the seven conditions of two-band noise-vocoded speech were measured for auditory (A), visual (V) and auditory–visual (AV) modalities after a few hours of bimodal (AV) training. The experiment was performed with 56 subjects with normal hearing. Performance of A and AV word recognition was significantly different under the seven auditory conditions. The V word intelligibility was not influenced by the condition of combined auditory cues. However, V word intelligibility was correlated with AV word recognition under all frequency conditions. Correlation between A and AV word intelligibilities was ambiguous. These findings suggest the importance of visual cues in AV speech perception under extremely degraded auditory conditions, and underscore the importance of the possible effectiveness of bimodal audio–visual training in postoperative rehabilitation for patients with postlingual deafness who have undergone artificial auditory device implantation.

Group Audiologic Rehabilitation for Adults and Their Communication Partners

Group Audiologic Rehabilitation for Adults and Their Communication Partners

Objective eye-gaze behaviour during face-to-face communication with proficient alaryngeal speakers: a preliminary study

There is a large body of research demonstrating the impact of visual information on speaker intelligibility in both normal and disordered speaker populations. However, there is minimal information on which specific visual features listeners find salient during conversational discourse. To investigate listeners' eye-gaze behaviour during face-to-face conversation with normal, laryngeal and proficient alaryngeal speakers. Sixty participants individually participated in a 10-min conversation with one of four speakers (typical laryngeal, tracheoesophageal, oesophageal, electrolaryngeal; 15 participants randomly assigned to one mode of speech). All speakers were > 85% intelligible and were judged to be 'proficient' by two certified speech-language pathologists. Participants were fitted with a head-mounted eye-gaze tracking device (Mobile Eye, ASL) that calculated the region of interest and mean duration of eye-gaze. Self-reported gaze behaviour was also obtained following the conversation using a 10 cm visual analogue scale. While listening, participants viewed the lower facial region of the oesophageal speaker more than the normal or tracheoesophageal speaker. Results of non-hierarchical cluster analyses showed that while listening, the pattern of eye-gaze was predominantly directed at the lower face of the oesophageal and electrolaryngeal speaker and more evenly dispersed among the background, lower face, and eyes of the normal and tracheoesophageal speakers. Finally, results show a low correlation between self-reported eye-gaze behaviour and objective regions of interest data. Overall, results suggest similar eye-gaze behaviour when healthy controls converse with normal and tracheoesophageal speakers and that participants had significantly different eye-gaze patterns when conversing with an oesophageal speaker. Results are discussed in terms of existing eye-gaze data and its potential implications on auditory-visual speech perception.

The effects of blurred vision on auditory-visual speech perception in younger and older adults

Speech understanding is improved when the observer can both see and hear the talker. This study compared the effects of reduced visual acuity on auditory-visual (AV) speech-recognition in noise among younger and older adults. Two groups of participants performed a closed-set sentence-recognition task in one auditory-alone (A-alone) condition and under three AV conditions: normal visual acuity (6/6), and with blurred vision to simulate a 6/30 and 6/60 visual impairment. The results showed that (1) the addition of visual speech cues improved speech-perception relative to the A-alone condition, (2) under the AV conditions, performance declined as the amount of blurring increased, (3) even under the AV condition that simulated a visual acuity of 6/60, the speech recognition scores were significantly higher than those obtained under the A-alone condition, and (4) generally, younger adults obtained higher scores than older adults under all conditions. Our results demonstrate the benefits of visual cues to enhance speech understanding even when visual acuity is not optimal.SumarioLa comprensión del lenguaje mejora cuando el observador puede tanto ver como oír a su interlocutor. Este estudio compara los efectos de una agudeza visual reducida, en el reconocimiento auditivo-visual (AV) del lenguaje con ruido, entre adultos jóvenes y mayores. Dos grupos de participantes realizaron una tarea de reconocimiento de oraciones en contexto cerrado en una condición solo auditiva (Solo-A) y en tres condiciones AV: con agudeza visual normal (6/6) y con visión borrosa simulando impedimentos visuales de 6/30 y 6/60. Los resultados mostraron: (1) la adición de claves visuales de lenguaje mejora la percepción del mismo en relación con la condición de Solo-A; (2) en las condiciones AV, el desempeño declinó conforme aumentó el grado de visión borrosa; (3) incluso en la condición AV que simulaba una agudeza visual de 6/60, las puntuaciones en el reconocimiento del lenguaje fueron significativamente superiores que las obtenidas en la condición de Solo-A y (4) generalmente, los adultos jóvenes obtuvieron puntuaciones más altas que los adultos mayores, en todas las condiciones. Nuestros resultados demuestran los beneficios de las claves visuales para aumentar la comprensión del lenguaje, incluso cuando la agudeza visual no es óptima.

Two-year-old phonology: impact of input, motor and cognitive abilities on development

Previous research has rarely compared the contributions of different underlying abilities to phonological acquisition. In this study, the auditory-visual speech perception, oro-motor and rule abstraction skills of 62 typically developing two-year olds were assessed and contrasted with the accuracy of their spoken phonology. Measures included auditory-visual speech perception, production of isolated and sequenced oro-motor movements, and verbal and non-verbal rule abstraction. Abilities in all three domains contributed to phonological acquisition. However, the use of atypical phonological rules was associated with lower levels of phonological accuracy and a linear regression indicated that this measure of rule abstraction had greater explanatory power than the measures of input processing and output skill.

Developmental factors and the non-native speaker effect in auditory-visual speech perception

The study investigated language and developmental factors in the use of visual information in audiovisual speech perception in speakers from different language backgrounds. Mandarin-Chinese and English adults and 8- to 9-year-old children were presented with /ba/, /da/, and /ga/ tokens spoken by two English and two Mandarin-Chinese speakers. A syllable identification task was presented in auditory-only, visual-only, and audiovisual (congruent and incongruent) conditions in clear and in noise. The results showed an increase in the use of visual information in adults relative to children in both the Chinese and English groups. In addition, a positive correlation between the total visual effect and speechreading performance was found, suggesting that the smaller visual influence in the bimodal condition for children might be accounted by their less sophisticated speechreading ability. In regard to the language factor, it was found that Chinese perceivers use visual information in their audiovisual speech processing to the same extent as English perceivers. Finally, there was evidence for a "non-native speaker effect" (i.e., stronger visual effect for non-native speech stimuli), but only for the English participants. Results from the current study suggest that the visual appearance of individual speakers and the acoustic-phonetic properties of specific languages should be considered in future cross-language studies.

Lip-Reading Aids Word Recognition Most in Moderate Noise: A Bayesian Explanation Using High-Dimensional Feature Space

Watching a speaker's facial movements can dramatically enhance our ability to comprehend words, especially in noisy environments. From a general doctrine of combining information from different sensory modalities (the principle of inverse effectiveness), one would expect that the visual signals would be most effective at the highest levels of auditory noise. In contrast, we find, in accord with a recent paper, that visual information improves performance more at intermediate levels of auditory noise than at the highest levels, and we show that a novel visual stimulus containing only temporal information does the same. We present a Bayesian model of optimal cue integration that can explain these conflicts. In this model, words are regarded as points in a multidimensional space and word recognition is a probabilistic inference process. When the dimensionality of the feature space is low, the Bayesian model predicts inverse effectiveness; when the dimensionality is high, the enhancement is maximal at intermediate auditory noise levels. When the auditory and visual stimuli differ slightly in high noise, the model makes a counterintuitive prediction: as sound quality increases, the proportion of reported words corresponding to the visual stimulus should first increase and then decrease. We confirm this prediction in a behavioral experiment. We conclude that auditory-visual speech perception obeys the same notion of optimality previously observed only for simple multisensory stimuli.

Can auditory and visual speech perception be trained within a group setting?

This study attempted to determine whether auditory-only and auditory-visual speech perception could be trained in a group format. A randomized controlled trial with at least 16 participants per group was completed. A training-only group completed at least 5 hr of group speech perception training; a training plus psychosocial group completed at least 5 hr of group speech perception training and psychosocial exercises; and a control group did not receive training. Evaluations were conducted before and after training and included analytic and synthetic measures of speech perception, hearing loss-related and generic quality of life scales, and a class evaluation form. No significant group changes were measured on any of the analytic auditory-only or auditory-visual measures of speech perception, yet the majority of training participants (regardless of training group) reported improvement in auditory and auditory-visual speech perception. The training participants demonstrated a significant reduction on the emotional subscale of the hearing loss-related quality of life scale, while the control participants did not demonstrate a change on this subscale. Benefits of group audiologic rehabilitation classes may not result from an actual improvement in auditory or visual speech perception abilities, but participants still perceive training in these areas as useful.

Cross-language auditory-visual speech perception development

Speech perception is auditory-visual (AV), but there is more visual information use by English than Japanese language perceivers (Sekiyama & Tohkura, 1993). This raises two questions: (1) when and (2) why do such differences emerge? In Experiment 1 with English- and Japanese-language children (6, 8, 11 years) and adults, AV speech perception (AVSP) was equivalent in English and Japanese 6-year-olds, significantly increased between 6 and 8 years in English- but not Japanese-language children, and remained stable over all ages for the Japanese. The origin of the Japanese-English adult difference in English language children's dramatic rise in AVSP between 6 and 8 years was investigated in Experiment 2. English-language children (5 to 8 years) and adults were tested for AVSP, reading, articulation, and language-specific speech perception (focus on native and away from non-native speech sounds, a known correlate of reading, Burnham, 2003). Regressions showed that language specific speech perception reliably predicted AVSP. It appears children use extra information (AVSP) in the face of challenging linguistic hurdles (learning to read), but only when the language is phonetically complex and the phoneme-to-grapheme conversion rules of the script are not transparent (English, but not Japanese).

Impact of language on development of auditory‐visual speech perception

The McGurk effect paradigm was used to examine the developmental onset of inter-language differences between Japanese and English in auditory-visual speech perception. Participants were asked to identify syllables in audiovisual (with congruent or discrepant auditory and visual components), audio-only, and video-only presentations at various signal-to-noise levels. In Experiment 1 with two groups of adults, native speakers of Japanese and native speakers of English, the results on both percent visually influenced responses and reaction time supported previous reports of a weaker visual influence for Japanese participants. In Experiment 2, an additional three age groups (6, 8, and 11 years) in each language group were tested. The results showed that the degree of visual influence was low and equivalent for Japanese and English language 6-year-olds, and increased over age for English language participants, especially between 6 and 8 years, but remained the same for Japanese participants. This may be related to the fact that English language adults and older children processed visual speech information relatively faster than auditory information whereas no such inter-modal differences were found in the Japanese participants' reaction times.