Audiovisual Interactions Research Articles

Significant audio-visual interaction for spatially congruent stimuli

Significant audio-visual interaction for spatially congruent stimuli

Time course of multisensory interactions during audiovisual speech perception in humans: a magnetoencephalographic study

During social interaction speech is perceived simultaneously by audition and vision. We studied interactions in the processing of auditory (A) and visual (V) speech signals in the human brain by comparing neuromagnetic responses to phonetically congruent audiovisual (AV) syllables with the arithmetic sum of responses to A and V syllables. Differences between AV and A+V responses were found bilaterally in the auditory cortices 150–200 ms and in the right superior temporal sulcus (STS) 250–600 ms after stimulus onset, showing that both sensory-specific and multisensory regions of the human temporal cortices are involved in AV speech processing. Importantly, our results suggest that AV interaction in the auditory cortex precedes that in the multisensory STS region.

Seeing to hear better: evidence for early audio-visual interactions in speech identification

Lip reading is the ability to partially understand speech by looking at the speaker's lips. It improves the intelligibility of speech in noise when audio-visual perception is compared with audio-only perception. A recent set of experiments showed that seeing the speaker's lips also enhances sensitivity to acoustic information, decreasing the auditory detection threshold of speech embedded in noise [J. Acoust. Soc. Am. 109 (2001) 2272; J. Acoust. Soc. Am. 108 (2000) 1197]. However, detection is different from comprehension, and it remains to be seen whether improved sensitivity also results in an intelligibility gain in audio-visual speech perception. In this work, we use an original paradigm to show that seeing the speaker's lips enables the listener to hear better and hence to understand better. The audio-visual stimuli used here could not be differentiated by lip reading per se since they contained exactly the same lip gesture matched with different compatible speech sounds. Nevertheless, the noise-masked stimuli were more intelligible in the audio-visual condition than in the audio-only condition due to the contribution of visual information to the extraction of acoustic cues. Replacing the lip gesture by a non-speech visual input with exactly the same time course, providing the same temporal cues for extraction, removed the intelligibility benefit. This early contribution to audio-visual speech identification is discussed in relationships with recent neurophysiological data on audio-visual perception.

Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception

We studied the interactions in neural processing of auditory and visual speech by recording event-related brain potentials (ERPs). Unisensory (auditory - A and visual - V) and audiovisual (AV) vowels were presented to 11 subjects. AV vowels were phonetically either congruent (e.g., acoustic /a/ and visual /a/) or incongruent (e.g., acoustic /a/ and visual /y/). ERPs to AV stimuli and the sum of the ERPs to A and V stimuli (A+V) were compared. Similar ERPs to AV and A+V were hypothesized to indicate independent processing of A and V stimuli. Differences on the other hand would suggest AV interactions. Three deflections, the first peaking at about 85 ms after the A stimulus onset, were significantly larger in the ERPs to A+V than in the ERPs to both congruent and incongruent AV stimuli. We suggest that these differences reflect AV interactions in the processing of general, non-phonetic, features shared by the acoustic and visual stimulus (spatial location, coincidence in time). The first difference in the ERPs to incongruent and congruent AV vowels peaked at 155 ms from the A stimuli onset. This and two later differences are suggested to reflect interactions at phonetic level. The early general AV interactions probably reflect modified activity in the sensory-specific cortices, whereas the later phonetic AV interactions are likely generated in the heteromodal cortices. Thus, our results suggest that sensory-specific and heteromodal brain regions participate in AV speech integration at separate latencies and are sensitive to different features of A and V speech stimuli.

Audiovisual interactions in music reading. A reaction times and event-related potentials study.

The general aim of this experiment was to investigate the processes involved in reading musical notation and to study the relationship between written music and its auditory representation. Our main interest was to determine if musicians can develop expectancies for plausible or implausible auditory events on the sole basis of the visual score. Results showed that musicians can clearly expect auditory endings on the basis of visual information. These findings enliven the discussion on the question of whether music reading is actually music perception.

Dynamics of audio-visual interactions in the guinea pig brain: an electrophysiological study.

Audio-visual interactions and their specifications, evaluated by bioelectrical activities, in guinea pigs are presented in this study. The difference potential, as the evidence of an interaction, was calculated by subtracting the sum of averaged potentials recorded in visual and auditory events from the averaged potential recorded in an event where two stimuli combined in the same sweep. Dynamic investigations have shown an interaction when auditory stimulus is applied 24 ms before and 201 ms after visual stimulation. Latency between the difference potential and auditory stimulus was stable. Directional investigations have shown that the interaction is not observed when auditory and/or visual stimulation is used ipsilaterally, according to the recording side.

Audio–visual integration in temporal perception

In situations of audio–visual interaction, research has generally found that audition prevails over vision in temporal perception, while vision is dominant over audition for spatial perception. Modality appropriateness to a given task generally determines the direction of this inter-modality effect. However, we found a reverse effect in some situations where a change in the frequency of visual stimuli was associated with a perceived change in the frequency of auditory stimuli. In our experiment, 12 participants were asked to judge the change in the frequency of visual and auditory stimuli using a visual flicker and auditory flutter stimuli. In some conditions either the auditory or the visual information was ambiguous. In addition to confirming the expected finding that a change in the frequency of the auditory stimuli induced a perceived change in the frequency of the visual stimuli, we found a new phenomenon. When ambiguous auditory temporal cues were presented, the change in the frequency of the visual stimuli was associated with a perceived change in the frequency of the auditory stimuli. This suggests that cross-modal asymmetry effects are influenced by the reliability of visual and auditory information as well as modality appropriateness.

Evidence of a visual-to-auditory cross-modal sensory gating phenomenon as reflected by the human P50 event-related brain potential modulation

We investigated the existence of a cross-modal sensory gating reflected by the modulation of an early electrophysiological index, the P50 component. We analyzed event-related brain potentials elicited by audiovisual speech stimuli manipulated along two dimensions: congruency and discriminability. The results showed that the P50 was attenuated when visual and auditory speech information were redundant (i.e. congruent), in comparison with this same event-related potential component elicited with discrepant audiovisual dubbing. When hard to discriminate, however, bimodal incongruent speech stimuli elicited a similar pattern of P50 attenuation. We concluded to the existence of a visual-to-auditory cross-modal sensory gating phenomenon. These results corroborate previous findings revealing a very early audiovisual interaction during speech perception. Finally, we postulated that the sensory gating system included a cross-modal dimension.

Sound-induced illusory flash perception: role of gamma band responses.

In the recently discovered sound-induced illusory flash phenomenon, a single flash accompanied with two auditory beeps is perceived as two flashes in a majority of trials. Here we asked what the neural substrates distinguishing illusion and no-illusion (i.e. perception of single flash) percepts are under identical stimulus configuration. Wavelet based method was used to analyze gamma band (> 30 Hz) responses in the event-related potential (ERP) signals recorded over visual cortical regions. We found: (i) significantly higher oscillatory and induced gamma band responses in illusion than in no-illusion trials, and (ii) significant supra-additive audio-visual interactions only in illusion trials. These results provide a clear neurophysiological correlate to the perception of illusion. Furthermore, the results suggest that auditory stimuli modulate cortical processing of visual stimuli, and the flash illusion (qualitative alteration of visual percept) only takes place when this modulation exceeds some critical threshold for the registration of conscious awareness.

Dynamics of cortico-subcortical cross-modal operations involved in audio-visual object detection in humans.

Very recently, a number of neuroimaging studies in humans have begun to investigate the question of how the brain integrates information from different sensory modalities to form unified percepts. Already, intermodal neural processing appears to depend on the modalities of inputs or the nature (speech/non-speech) of information to be combined. Yet, the variety of paradigms, stimuli and technics used make it difficult to understand the relationships between the factors operating at the perceptual level and the underlying physiological processes. In a previous experiment, we used event-related potentials to describe the spatio-temporal organization of audio-visual interactions during a bimodal object recognition task. Here we examined the network of cross-modal interactions involved in simple detection of the same objects. The objects were defined either by unimodal auditory or visual features alone, or by the combination of the two features. As expected, subjects detected bimodal stimuli more rapidly than either unimodal stimuli. Combined analysis of potentials, scalp current densities and dipole modeling revealed several interaction patterns within the first 200 micro s post-stimulus: in occipito-parietal visual areas (45-85 micro s), in deep brain structures, possibly the superior colliculus (105-140 micro s), and in right temporo-frontal regions (170-185 micro s). These interactions differed from those found during object identification in sensory-specific areas and possibly in the superior colliculus, indicating that the neural operations governing multisensory integration depend crucially on the nature of the perceptual processes involved.

Visual illusion induced by sound

We present the first cross-modal modification of visual perception which involves a phenomenological change in the quality—as opposed to a small, gradual, or quantitative change—of the percept of a non-ambiguous visual stimulus. We report a visual illusion which is induced by sound: when a single flash of light is accompanied by multiple auditory beeps, the single flash is perceived as multiple flashes. We present two experiments as well as several observations which establish that this alteration of the visual percept is due to cross-modal perceptual interactions as opposed to cognitive, attentional, or other origins. The results of the second experiment also reveal that the temporal window of these audio–visual interactions is approximately 100 ms.

Audio-visual integration in schizophrenia

Integration of information provided simultaneously by audition and vision was studied in a group of 18 schizophrenic patients. They were compared to a control group, consisting of 12 normal adults of comparable age and education. By administering two tasks, each focusing on one aspect of audio-visual integration, the study could differentiate between a spatial integration deficit and a speech-based integration deficit. Experiment 1 studied audio-visual interactions in the spatial localisation of sounds. Experiment 2 investigated integration of auditory and visual speech. The schizophrenic group performed as the control group on the sound localisation task, but in the audio-visual speech task, there was an impairment in lipreading as well as a smaller impact of lipreading on auditory speech information. Combined with findings about functional and neuro-anatomical specificity of intersensory integration, the data suggest that there is an integration deficit in the schizophrenic group that is related to the processing of phonetic information.

Content-based video parsing and indexing based on audio-visual interaction

A content-based video parsing and indexing method is presented in this paper, which analyzes both information sources (auditory and visual) and accounts for their inter-relations and synergy to extract high-level semantic information. Both frame- and object-based access to the visual information is employed. The aim of the method is to extract semantically meaningful video scenes and assign semantic label(s) to them. Due to the temporal nature of video, time has to be accounted for. Thus, time-constrained video representations and indices are generated. The current approach searches for specific types of content information relevant to the presence or absence of speakers or persons. Audio-source parsing and indexing leads to the extraction of a speaker label mapping of the source over time. Video-source parsing and indexing results in the extraction of a talking-face shot mapping over time. Integration of the audio and visual mappings constrained by interaction rules leads to higher levels of video abstraction and even partial detection of its context.

Rules of Intersensory Integration in Spatial Scene Analysis and Speechreading

Audiovisual interactions observed in the ventriloquism and McGurk illusions were compared, using the same materials. The ventriloquism effect was estimated through a discordance detection task and the McGurk illusion through an identification task. The stimuli were visually displayed on a screen located in front of the participants’ head and acoustically delivered through one of nine hidden loudspeakers located from straight ahead (0°) to 80° (20° steps) left and right. The speaker’s face was either upright or inverted. The ventriloquism effect was affected by the degree of spatial separation, but unaffected by upright vs. inverted presentation of the face, or by the congruency of the stimuli. The McGurk illusion was of the same size whatever the loudspeaker location but was reduced by face inversion. The differences in the spatial and cognitive rules that govern both interactions are discussed in terms of specific functionality of the underlying mechanisms.

Investigation of the influence of visual stimuli on the breakdown of the echo threshold

Audiovisual interaction is already known from the ventriloquism effect [W. R. Thurlow and Ch. E. Jack, Percept. Mot. Skills 36, 1171–1174 (1973)]. Synchronously presented auditory and visual stimuli are perceptually grouped and localized as one object. The performed experiments aimed to explore a possible influence of visual stimuli on echo suppression. They focused on investigating whether visual stimuli can also cause a breakdown of the echo threshold. One hypothesis is that the breakdown of echo suppression is caused by violations of the listeners’ expectations concerning the presented scenario. The question is whether this implausibility can be attributed to acoustical reasons. Two different kinds of visual implausibilities were tested by measuring echo threshold with an adaptive paradigm. In the build-up phase of both experiments, visual and auditory parts of the stimuli have been presented from the same direction. In the test phase, the visual stimulus’ direction was altered. The main result of the investigations was that both experiments did not provide any evidence of a visually induced breakdown of the echo threshold. This holds true, even if the position of the auditory event corresponding to the direct sound was changed due to the ventriloquism effect by changing the corresponding visual stimulus’ direction of incidence.

A sound judgment depending on the urban visual setting?

In a number of experiments carried out in the experimental psychology field, audition and vision proved to provide information which interacted each other. What about the audio–visual interactions in the city? The first aim of this research was to develop an experimental procedure involving methodological parameters best suitable to test the influence of the visual setting on the sound judgment in the complex sound environment. The second one was to apply it to various urban situations. In a series of simulation tests, participants rated some urban sound stimuli and this, under different visual conditions varying according to the degree of urbanization. An artificial audio–visual environment was specifically created. The experimental methods were based on three types, involving different structures of presentation of the audio–visual combinations. For each of them (except for the completely random one), two types of orders of presentation of both the visual and sound sequences were defined. The results for all the experiments were commented on and compared. The best suitable experimental method was validated. The present results point out that the visual influence was many sided. Some clusters of sound stimuli and sound variables were defined according to the visual influence exerted.

Influence of visual attention upon sound localization

Previous studies on audio-visual interaction such as the ventriloquism effect have indicated cognitive (or context) factors as well as sensory (or synchronous) factors could make the interaction. In these studies, however, visual attention seems to have been neglected. Thus, it has been still unknown whether the visual attention affects the interaction or not. We investigated the contribution of the attention factors to the audio-visual interaction by comparing the sound localization biases made by attentional factors and those by synchronous factors. Three subjects participated in the localization tasks in horizontal and vertical orientations. As the results, we found small influence of the attentional factors upon the interaction in the horizontal orientation, and no influence in the vertical orientation. On the contrary, the effect of the synchronous factors was larger in the vertical orientation than in the horizontal orientation. We concluded that the visual attention could affect the audio-visual interaction slightly, and that the influences of the attentional factors and the perceptual factors upon the interactions were made in the different processes.

Speeded responses to audiovisual signal changes result from bimodal integration

Integration of auditory and visual information was studied in humans detecting targets (i.e., location changes of the auditory, the visual, or both parts of a repetitively presented audiovisual stimulus). Behavioral results suggest that the time advantage to bimodal compared with unimodal targets was due to combined rather than separate processing of the auditory and the visual target information. Event-related brain potential results revealed strong audiovisual interactions supporting interactive and not independent coactivation models. The time course of this interaction suggests that the audiovisual integration occurred after low-level, sensory processing but well before the execution of the motor response.

Rural telepsychiatry is economically unsupportable: the Concorde crashes in a cornfield.

T the use of communication technologies to offer psychiatric services, has become a topic of great interest as a means of overcoming the effects of the shortage of psychiatrists in rural areas (1–4). The idea of using telecommunication in psychiatry is not new. In fact, examples can be found as far back as 1959, and the topic has been well reviewed (1,4–6). Although the technology has improved to the point of providing adequate audiovisual interactions, the cost-effectiveness of these interactions remains unassessed. Clearly, technological feasibility does not eliminate the need for economic practicality. Before relying on telepsychiatry as a solution to inadequate psychiatric services in rural areas, we need to consider economic variables such as the cost of the equipment, charges for maintaining connections, the volume of use, and reimbursement. We also need to consider its impact on existing systems of care into which it would be introduced. We studied the feasibility of implementing a telepsychiatry system to link psychiatrists at Michigan State University with patients at a community mental health center (CMHC) in rural Michigan. The target county we studied has a population of 58,000 and is approximately 75 miles from the university. The county is poor and demographically similar to many rural counties and is designated by the U.S. Department of Health and Human Services as an area with a shortage of mental health professionals. Many people lack transportation. Based on our feasibility study, this column examines the economics of setting up a telepsychiatry system through which care can be provided to a rural area. Moreover, we estimated the cost of delivering one type of patient care (medication management) via this system. The purpose of the paper is not to provide a technological discussion of designing a system. Rather, we offer an analysis of economic issues and systems problems encountered in setting up a very basic telepsychiatry installation at the low end of the cost range.

Complementarity and synergy in bimodal speech: Auditory, visual, and audio-visual identification of French oral vowels in noise

The efficacy of audio-visual interactions in speech perception comes from two kinds of factors. First, at the information level, there is some "complementarity" of audition and vision: It seems that some speech features, mainly concerned with manner of articulation, are best transmitted by the audio channel, while some other features, mostly describing place of articulation, are best transmitted by the video channel. Second, at the information processing level, there is some "synergy" between audition and vision: The audio-visual global identification scores in a number of different tasks involving acoustic noise are generally greater than both the auditory-alone and the visual-alone scores. However, these two properties have been generally demonstrated until now in rather global terms. In the present work, audio-visual interactions at the feature level are studied for French oral vowels which contrast three series, namely front unrounded, front rounded, and back rounded vowels. A set of experiments on the auditory, visual, and audio-visual identification of vowels embedded in various amounts of noise demonstrate that complementarity and synergy in bimodal speech appear to hold for a bundle of individual phonetic features describing place contrasts in oral vowels. At the information level (complementarity), in the audio channel the height feature is the most robust, backness the second most robust one, and rounding the least, while in the video channel rounding is better than height, and backness is almost invisible. At the information processing (synergy) level, transmitted information scores show that all individual features are better transmitted with the ear and the eye together than with each sensor individually.