Auditory-visual Integration Research Articles

Discrimination and labeling differences in audiovisual spech perception

In an earlier report, Clement et al. [J. Acoust. Soc. Am. 107, 2887 (2000)] showed that listeners’ confidence ratings of the identity of audiovisual syllables produced with conflicting auditory and visual cues were typically lower than those for audiovisually congruent syllables. This occurred even when both types of syllables were labeled consistently as the same phoneme. Confidence ratings varied across listeners, suggesting a graded nature of auditory–visual integration even across listeners who exhibit visual bias. In the current experiment, listeners were asked to identify audiovisual tokens from two talkers who elicit a high proportion of visually biased responses [Carney et al., J. Acoust. Soc. Am. 106, 2270 (1999)] and to rate their confidence in their labeling judgment. Only listeners who exhibited strong visual bias were tested in a discrimination task in which a true audiovisual /di/ was a standard. Comparison stimuli were visual /gi/-auditory /bi/, visual /bi/-auditory /gi/, and true audiovisual /bi/, /di/, and /gi/. Listeners were able to discriminate the visually biased and true /di/ stimuli while labeling them both as alveolar tokens. Results support the notion of a graded perception of bimodal stimuli. [Research supported by NIDCD.]

Spatio-temporal constraints for auditory–visual integration

The perceptual coherence of auditory and visual information is achieved by integrative brain processes. Specialized single neurons with spatial and temporal interactions of auditory and visual stimuli have been demonstrated by several neurophysiological studies. The present, psychophysical, study investigates possible perceptual correlates of these neuronal features. Subjects had to indicate the point of subjective spatial alignment (PSSA) for a horizontally moving visual stimulus that crossed the position of a stationary sound source. Auditory and visual stimuli consisted of periodic pulses that were systematically varied in their phase relationship or repetition rate. PSSAs obtained for continuous visual stimuli served as a reference. When sound and light pulses were coincident in phase at a repetition rate of 2 Hz, PSSAs were shifted by ∼3° in a direction opposite to the movement of the visual stimulus (with respect to the reference condition). This shift markedly decreased when the temporal disparity exceeded ∼100 ms and disappeared near phase opposition (250 ms disparity). With 4 Hz repetition rate (temporal disparity ≤125 ms), there was no significant effect of phase relationship on PSSAs, but still an approximately constant shift with respect to the reference value. Variation of the repetition rate resulted in almost constant shifts in PSSA of ∼3° between 1 and 4 Hz and a linear decrease (slope 0.27°/Hz) with higher repetition rates. These results suggest a spatio-temporal ‘window’ for auditory–visual integration, that extends over ∼100 ms and ∼3°: when auditory and visual stimuli are within this window, they are always perceived as spatially coincident. These psychophysical findings may be related to properties of bimodal neurons such as have been demonstrated by neurophysiological recordings in midbrain and cortex.

Tests of auditory-visual integration efficiency within the framework of the fuzzy logical model of perception.

The fuzzy logical model of perception [FLMP, Massaro, Perceiving Talking Faces: From Speech Perception to a Behavioral Principle (MIT Press, Cambridge, MA, 1998)] has been extremely successful at describing performance across a wide range of ecological domains as well as for a broad spectrum of individuals. Because the model predicts optimal or maximally efficient integration, an important issue is whether this is the case for most individuals. Three databases are evaluated to determine to what extent a significant quantitative improvement in predictive ability can be obtained if integration is assumed to be somewhat inefficient. For the most part, there were no significant signs of inefficient integration. The previous differences found by Grant and Seitz [J. Acoust. Soc. Am. 104, 2438-2450 (1998)] must be due to their measures of efficiency, which appear to be invalid and/or conflate information with integration efficiency. Finally, the descriptive ability of the FLMP is shown to be theoretically informative and not simply the model's ability to describe any possible outcome.

Auditory–visual integration of talker gender in vowel perception

The experiments reported here used auditory–visual mismatches to compare three approaches to speaker normalization in speech perception: radical invariance, vocal tract normalization, and talker normalization. In contrast to the first two, the talker normalization theory assumes that listeners' subjective, abstract impressions of talkers play a role in speech perception. Experiment 1 found that the gender of a visually presented face affects the location of the phoneme boundary between [Ω] and [Λ] in the perceptual identification of a continuum of auditory–visual stimuli ranging from hood to hud. This effect was found for both “stereotypical” and “non-stereotypical” male and female voices. The experiment also found that voice stereotypicality had an effect on the phoneme boundary. The difference between male and female talkers was greater when the talkers were rated by listeners as “stereotypical”. Interestingly, for the two female talkers in this experiment, rated stereotypicality was correlated with voice breathiness rather than vowel fundamental frequency. Experiment 2 replicated and extended experiment 1 and tested whether the visual stimuli in experiment 1 were being perceptually integrated with the acoustic stimuli. In addition to the effects found in experiment 1, there was a boundary effect for the visually presented word: listeners responded hood more frequently when the acoustic stimulus was paired with a movie clip of a talker saying hood. Experiment 3 tested the abstractness of the talker information used in speech perception. Rather than seeing movie clips of male and female talkers, listeners were instructed to imagine a male or female talker while performing an audio-only identification task with a gender-ambiguous hood-hud continuum. The phoneme boundary differed as a function of the imagined gender of the talker. The results from these experiments suggest that listeners integrate abstract gender information with phonetic information in speech perception. This conclusion supports the talker normalization theory of perceptual speaker normalization.

Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study.

The aim of this study was (1) to provide behavioral evidence for multimodal feature integration in an object recognition task in humans and (2) to characterize the processing stages and the neural structures where multisensory interactions take place. Event-related potentials (ERPs) were recorded from 30 scalp electrodes while subjects performed a forced-choice reaction-time categorization task: At each trial, the subjects had to indicate which of two objects was presented by pressing one of two keys. The two objects were defined by auditory features alone, visual features alone, or the combination of auditory and visual features. Subjects were more accurate and rapid at identifying multimodal than unimodal objects. Spatiotemporal analysis of ERPs and scalp current densities revealed several auditory-visual interaction components temporally, spatially, and functionally distinct before 200 msec poststimulus. The effects observed were (1) in visual areas, new neural activities (as early as 40 msec poststimulus) and modulation (amplitude decrease) of the N185 wave to unimodal visual stimulus, (2) in the auditory cortex, modulation (amplitude increase) of subcomponents of the unimodal auditory N1 wave around 90 to 110 msec, and (3) new neural activity over the right fronto-temporal area (140 to 165 msec). Furthermore, when the subjects were separated into two groups according to their dominant modality to perform the task in unimodal conditions (shortest reaction time criteria), the integration effects were found to be similar for the two groups over the nonspecific fronto-temporal areas, but they clearly differed in the sensory-specific cortices, affecting predominantly the sensory areas of the nondominant modality. Taken together, the results indicate that multisensory integration is mediated by flexible, highly adaptive physiological processes that can take place very early in the sensory processing chain and operate in both sensory-specific and nonspecific cortical structures in different ways.

Measures of auditory-visual integration in nonsense syllables and sentences.

For all but the most profoundly hearing-impaired (HI) individuals, auditory-visual (AV) speech has been shown consistently to afford more accurate recognition than auditory (A) or visual (V) speech. However, the amount of AV benefit achieved (i.e., the superiority of AV performance in relation to unimodal performance) can differ widely across HI individuals. To begin to explain these individual differences, several factors need to be considered. The most obvious of these are deficient A and V speech recognition skills. However, large differences in individuals' AV recognition scores persist even when unimodal skill levels are taken into account. These remaining differences might be attributable to differing efficiency in the operation of a perceptual process that integrates A and V speech information. There is at present no accepted measure of the putative integration process. In this study, several possible integration measures are compared using both congruent and discrepant AV nonsense syllable and sentence recognition tasks. Correlations were tested among the integration measures, and between each integration measure and independent measures of AV benefit for nonsense syllables and sentences in noise. Integration measures derived from tests using nonsense syllables were significantly correlated with each other; on these measures, HI subjects show generally high levels of integration ability. Integration measures derived from sentence recognition tests were also significantly correlated with each other, but were not significantly correlated with the measures derived from nonsense syllable tests. Similarly, the measures of AV benefit based on nonsense syllable recognition tests were found not to be significantly correlated with the benefit measures based on tests involving sentence materials. Finally, there were significant correlations between AV integration and benefit measures derived from the same class of speech materials, but nonsignificant correlations between integration and benefit measures derived from different classes of materials. These results suggest that the perceptual processes underlying AV benefit and the integration of A and V speech information might not operate in the same way on nonsense syllable and sentence input.

Auditory-visual speech recognition by hearing-impaired subjects: consonant recognition, sentence recognition, and auditory-visual integration.

Factors leading to variability in auditory-visual (AV) speech recognition include the subject's ability to extract auditory (A) and visual (V) signal-related cues, the integration of A and V cues, and the use of phonological, syntactic, and semantic context. In this study, measures of A, V, and AV recognition of medial consonants in isolated nonsense syllables and of words in sentences were obtained in a group of 29 hearing-impaired subjects. The test materials were presented in a background of speech-shaped noise at 0-dB signal-to-noise ratio. Most subjects achieved substantial AV benefit for both sets of materials relative to A-alone recognition performance. However, there was considerable variability in AV speech recognition both in terms of the overall recognition score achieved and in the amount of audiovisual gain. To account for this variability, consonant confusions were analyzed in terms of phonetic features to determine the degree of redundancy between A and V sources of information. In addition, a measure of integration ability was derived for each subject using recently developed models of AV integration. The results indicated that (1) AV feature reception was determined primarily by visual place cues and auditory voicing + manner cues, (2) the ability to integrate A and V consonant cues varied significantly across subjects, with better integrators achieving more AV benefit, and (3) significant intra-modality correlations were found between consonant measures and sentence measures, with AV consonant scores accounting for approximately 54% of the variability observed for AV sentence recognition. Integration modeling results suggested that speechreading and AV integration training could be useful for some individuals, potentially providing as much as 26% improvement in AV consonant recognition.

Fusion of experimental data

The integration of information from various sensory systems is one of the most difficult challenges in understanding both perception and cognition. For example, the problem of auditory-visual integration is a correspondence problem between perceived auditory and visual scenes. Two main questions arise when designing data analysis systems: what is the useful information to be integrated?, and what are the integration rules? The problem of integrating information becomes relevant whenever: (a) the same kind of data are detected by spatially distributed sensors; (b) heterogeneous data are detected by different sensors; (c) heterogeneous distributed data are involved. General problems concerning the integration of experimental data are reviewed. The case of the BeppoSAX X-ray astronomical satellite is given as an example.

Chapter 12 Visual-auditory integration in cat superior colliculus: implications for neuronal control of the orienting response

Previous physiological studies have demonstrated that inputs from different sensory modalities converge on individual neurons in the superior colliculus. Moreover, in anesthetized, paralyzed animals, those tectal neurons which are most directly connected to brain stem circuits mediating orienting eye and head movements are highly likely to exhibit significant integration of sensory inputs from multiple modalities. The purpose of the present study was to examine the responses of tectal neurons in the alert cat when visual and auditory stimuli were presented as targets for ocular fixation and orienting responses. For comparison to previous work in anesthetized, paralyzed animals, we also examined the responses of tectal neurons to the presentation of these stimuli during periods when the cats voluntarily maintained their eyes near primary position in the absence of a fixation target. Under these conditions, there were significant differences between the strength of the response to the simultaneous presentation of visual and auditory targets and the strength of response to the most effective unimodal stimulus in about 40% of the cells tested. Many tectal neurons also responded tonically during fixation of visual, auditory and bimodal targets, and some of these also exhibited significant bimodal interactions. However, among individual neurons which responded phasically to stimulus onset or offset and tonically during fixation, there was only a weak correlation between the extent of bimodal interaction under the two conditions. Finally, among saccade-related neurons, the magnitude of saccade-related activity was only slightly affected when a biomodal target was used to elicit a saccade, and the extent of bimodal interactions was generally less than was found for the onset and offset of sensory targets. Such multisensory interactions can be significant for behavior. Indeed, simply using a multisensory target has been shown to influence the probability and latency of overt orienting responses, although the extent of such effects will probably vary across both tasks and stimulus conditions. Strong multi-sensory interactions are most likely to occur when low intensity stimuli are used. Our use of moderately intense sensory stimuli probably accounts for our finding of a relatively small percentage of cells in which bimodal responses were greater than the sum of their unimodal responses.

Measures of auditory‐visual integration

The easiest and most effective way to improve speech recognition for hearing‐impaired individuals, or for normal‐hearing individuals listening in noisy or reverberant environments, is to have them watch the talker’s face. Auditory‐visual (AV) speech recognition has been shown consistently to be better than either hearing alone or speechreading alone for all but the most profoundly hearing‐impaired individuals. When AV recognition is less than perfect, several factors need to be considered. The most obvious of these are poor auditory (A) and poor visual (V) speech recognition skills. However, even when differences in unimodal skill levels are taken into account, differences among individual AV recognition scores persist. At least part of these individual differences may be attributable to differing abilities to integrate A and V cues. Unfortunately, there is no widely accepted measure of AV integration ability. Recent models of AV integration offer a quantitative means for estimating individual integration abilities for phoneme recognition. In this study, we compare several possible integration measures, along with model predictions, using both congruent and discrepant AV phoneme and sentence recognition tasks. The focus of this talk will address the need for independent measures of AV integration for individual subjects. [Work supported by NIH Grant DC00792.]

Experiments on the Auditory and Vocal Functions in the Chimpanzee

Experiments on the auditory and vocal functions of the chimpanzee were reported. The first main focus was on the basic auditory functions such as auditory sensitivity and difference thresholds, on the auditory cognition such as auditory-visual intermodal integration, and on the speech perception such as the prototype and context effects. A chimpanzee showed difficulties in the auditory-visual inter-modal integration. The second focus was on the close auditory-vocal interrelationship. It was reported that the chimpanzee had a W-shaped auditory sensitivity. The perception experiments showed that the second formant was not important for the perception of human vowels and vowel-like vocal sounds of the chimpanzee. The acoustic analyses showed that the second formant was constant between these vowel-like vocal sounds. These results indicated that the auditory and vocal functions were closely inter-related each other. The third focus was on the early vocal development of an infant chimpanzee. A comparison in vocal development between chimpanzee and human infant suggested that the chimpanzee infant did not reach to the Oller's Expansion Stage of the human vocal development. The high larynx and the small pharinx were suggested to be partially responsible for the limitation of the vocal development and the repertoire of vowel-like vocal sounds in the chimpanzee.

The transmission of prosodic information via selected spectral regions of speech

In a recent article [K. W. Grant and L. D. Braida, J. Acoust. Soc. Am. 89, 2952–2960 (1991)] it was demonstrated that spectrally different bands of speech with equal articulation index (AI) scores provided approximately equal auditory-visual sentence recognition when combined with speechreading. Given that different parts of the frequency spectrum provide different segmental cues for consonant and vowel recognition, current models of auditory-visual integration [e.g., L. D. Braida, J. Exp. Psychol. 43A, 647–677 (1991)] would predict that some spectral regions of speech are more complementary to speechreading than others. This raises the possibility that the findings of Grant and Braida are attributable, at least in part, to suprasegmental cues that were transmitted equally well by the different spectral bands tested. To test this possibility, the identification of syllable number, syllable stress, sentence intonation, and phrase boundary location was assessed under six approximately equal AI filter conditions similar to those evaluated by Grant and Braida. The results indicate that syllable number and syllable stress are perceived best through high-frequency bands, intonation is perceived best through low-frequency bands, and phrase boundary location is perceived equally well throughout the speech spectrum. These results are discussed in terms of the importance of different spectral regions of speech for the recognition of suprasegmental cues, and how this may relate to overall speech intelligibility. [Work supported by NIH Grant No. DC00792.]

Visual-auditory integration and vocalic effects on fricative perception.

This study investigates the effect of visual presentation of four syllables ([sa], [sha], [su], [shu]) on perception of a fricative noise continuum. The visual stimuli were taken from videotapes of a speaker uttering the four syllables. The nine-step noise continuum ranged from [s] to [sh], and when followed by the acoustic vocalic portions taken from natural utterances of [sa], etc., replicated Mann and Repp’s (1980) findings about the context effects of vowel lip rounding and consonant place of articulation on fricative perception. To determine if similar context effects would occur when the vocalic information was specified visually rather than auditorily, stimuli from the noise continuum were dubbed in place of the natural utterances on the video tapes and subjects identified the visual-auditory hybrids by naming each syllable aloud. The results showed a significant lip rounding effect on perception of ‘‘s’’ vs ‘‘sh’’ indicating that listeners’ compensation for the decrease in frication noise frequency that is caused by anticipatory lip rounding takes place whether the lip rounding is perceived visually or auditorily. The effect of consonant place of articulation was less consistent.

The Staggered Spondaic Word Test

We have described three major groupings that encompass most auditory processing difficulties. While the problems may be superimposed upon one another in any individual client, each diagnostic sign is closely associated with particular communication and learning disorders. In addition, these behaviors may be related back to the functional anatomy of the regions that are implicated by the SSW test. The auditory-decoding group is deficient in rapid analysis of speech. The vagueness of speech sound knowledge is thought to lead to auditory misunderstanding and confusion. In early life, this may be reflected in the child's articulation. Poor phonic skills that result from this deficit are thought to contribute to their limited reading and spelling abilities. The auditory tolerance-fading memory group is often thought to have severe auditory-processing problems because those in it are highly distracted by background sounds and have poor auditory memories. However, school performance is not far from grade level, and the resulting reading disabilities stem more from limited comprehension than from an inability to sound out the words. Distractibility and poor auditory memory could contribute to the apparent weakness in reading comprehension. Many of the characteristics of the auditory tolerance-fading memory group are similar to those of attention deficit disorder cases. Both groups are associated anatomically with the AC region. The auditory integration cases can be divided into two subgroups. In the first, the subjects exhibit the most severe reading and spelling problems of the three major categories. These individuals closely resemble the classical dyslexics. We presume that this disorder represents a major disruption in auditory-visual integration. The second subgroup has much less severe learning difficulties, which closely follow the pattern of dysfunction of the auditory tolerance-fading memory group. The excellent physiological procedures to which we have been exposed during this Windows on the Brain conference provide a glimpse of the exciting possibilities for studying brain function. However, in working with individuals who have cognitive impairments, the new technology should be validated by standard behavioral tests. In turn, the new techniques will provide those who use behavioral measures with new parameters and concepts to broaden our understanding. For the past quarter of a century, the SSW test has been compared with other behavioral, physiological, and anatomical procedures. Based on the information that has been assembled, we have been able to classify auditory processing disorders into three major categories.(ABSTRACT TRUNCATED AT 400 WORDS)

Auditory-visual temporal-spatial integration abilities and academic performance for learning disordered children

Auditory-visual temporal-spatial integration abilities and academic performance for learning disordered children

Auditory-visual temporal-spatial integration abilities and academic performance for learning disordered children

Auditory-visual temporal-spatial integration abilities and academic performance for learning disordered children

The development of reading: as you seek so shall you find.

In this paper we critically examine different methods of investigation of reading development and reading disability. We argue that there is need for longitudinal studies of reading where a differential design allows assessment of the relative degree of involvement of associated skills. We then describe one study of this type. The results chart the development of children who become skilled readers and of those with generalized or specific reading disability. They also demonstrate the crucial effects of decisions regarding subject matching, test design, and measurement of differential abilities on the outcomes of such investigations. A cohort of 40 children was assessed for their abilities on 44 variables which involved reading, spelling, vocabulary, short-term memory (STM), visual skills, auditory-visual integration, language knowledge, rote knowledge and ordering ability as they developed from five to eight years old. Three groups were extracted at age eight on the basis of reading and IQ scores. Group A showed a relatively specific reading disability (high IQ, low reading), Group B were good readers of similarly high IQ, Group C showed a more generalized reading deficit whereby they were at the same level as Group A in reading but their IQ scores were low. The data were then searched retrospectively to describe the development of these patterns of ability from the very beginnings of reading acquisition. The children with specific reading retardation differed from their better-reading peers in terms of relatively few variables which concerned phonological segmentation, STM and naming. The children with generalized reading disability differed from their better-reading peers in almost every respect, but the strong discriminators concerned phonological processing. The children with specific reading disability differed from those with generalized reading disability in terms of abilities which involve visual processing. These patterns of ability were replicated at each age from five to seven years old.

The relationship of perceptual-motor development to learning readiness in kindergarten: a multivariate analysis.

In this paper an effort is made to bridge the gap between some of the more basic experimental research and theoretical concepts in cognitive psychology and information processing on the one hand and applied clinical research on the other hand. Prior studies have revealed significant linear relationships between independent visual functions and school readiness, however, the additive nature of perceptual-motor factors and their correlation with reading ability often has been neglected. Tests using tachistoscopic exposures with 2 digits at 0.1 sec. and 0.01 sec., the Divided Form Board, and the Grooved Pegboard, dominant and nondominant hand, all showed significant correlations with readiness in a population of 48 kindergarten children. The results of the Auditory-Visual Integration Test were not significant at the kindergarten level. To increase the strength of the relationship between perceptual-motor development and readiness, three factors were considered simultaneously in a multiple regression scheme: Tachistoscopic exposure of 2 digits at 0.01 seconds, Divided Form Board, and Grooved Pegboard, dominant hand. The resulting correlation supports the hypothesis that an individual whose perceptual skills are good is likely to have a high readiness score. The corresponding r2 confirmed that variations in perceptual skills explain more than 50% of the variations in learning readiness in this study.

Relations Between Spelling and Performance in Nigerian Elementary School Children

Abstract Correlates of written language ability were examined in 36 Nigerian elementary school children. Tests administered included a spelling test, the Goodenough Draw-A-Man intelligence test, a digit span task, a test of visual memory, visual-auditory and auditory-visual integration tests, and a visuo-motor perception task. The results showed that spelling correlated positively with school performance in Grades 3 and 4, with visual memory in Grades 2 and 4 and with the Draw-A-Man test in the fourth grade and showed visual-auditory integration in Grades 2 and 3 and auditory-visual integration in the second grade. These findings, which are consistent with those reported in the West, indicate that at different grade levels different processes are correlated with written language skills.

Auditory-Visual and Temporal-Spatial Pattern Matching Performance of Two Types of Learning-Disabled Children

Normal readers, learning-disabled readers with a visual processing dysfunction, and learning-disabled readers with an auditory processing dysfunction in grades two through four were compared on the following six pattern-matching tasks: auditory-temporal to visual-spatial (At-Vs), auditory-temporal to visual-temporal (At-Vt), visual-temporal to visual spatial (Vt-Vs), auditory-temporal to auditory-temporal (At-At), visual-temporal to visual-temporal (Vt-Vt), and visual spatial to visual-spatial (Vs-Vs). The results indicated a significant difference between the normal readers and the learning-disabled readers on the At-Vs and At-Vt tasks but not on the Vt-Vs task, which was interpreted in terms of learning-disabled readers being deficient in auditory-visual integration rather than temporal-spatial integration skills. A significant effect of learning disability type was also identified on the Vt-Vt and Vs-Vs tasks but not on the At-At or Vt-Vs tasks. These results were discussed in terms of differences in memory. Implications of the results for previous and future studies were discussed as well as possible practical implications for diagnosing modality strengths and instructing learning-disabled readers.