Audiovisual Binding Research Articles

Multistability, cross-modal binding and the additivity of conjoined grouping principles

We present a sceptical view of multimodal multistability--drawing most of our examples from the relation between audition and vision. We begin by summarizing some of the principal ways in which audio-visual binding takes place. We review the evidence that unambiguous stimulation in one modality may affect the perception of a multistable stimulus in another modality. Cross-modal influences of one multistable stimulus on the multistability of another are different: they have occurred only in speech perception. We then argue that the strongest relation between perceptual organization in vision and perceptual organization in audition is likely to be by way of analogous Gestalt laws. We conclude with some general observations about multimodality.

New evidence for audiovisual speech scene analysis: Low level interaction between auditory streaming and visual cues in speech perception

We have proposed in the last years that there could exist a level of audiovisual binding (audiovisual speech scene analysis) previously to audiovisual fusion and speech comprehension. In this paper, we propose a new paradigm in which a competition exists between a non-speech auditory stream and an audiovisual speech stream for incorporating the pre-voicing component (PVC) excised from a /b/. The auditory stream is composed of low intensity PVCs at a regular rhythm of about 1Hz, interleaved with /pa/ syllables at about 0.3Hz, forming the speech stream. The listeners' tasks is an online forced choice between /p/ and /b/, which relies on the level of capture of a PVC within the speech stream (hence perceived as /b/) or within the auditory stream (speech being hence perceived as /pa/). The regular PVC stream has the tendency to capture the common PVC. Surprisingly, the vision of lips movements enhances this capture effect, and the phonetic fusion between the common PVC and the /p/ is disfavored. To characterize this effect, two supplementary experiments have been carried out, suggesting this effect is (at least partly) speech specific. This supports our proposal for an audiovisual speech binding mechanism.

The size of the ventriloquist effect is modulated by emotional valence

It is currently unknown to what extent the integration of inputs from different modalities are subject to the influence of attention, emotion, and/or motivation. The ventriloquist effect is widely assumed to be an automatic, crossmodal phenomenon, normally shifting the perceived location of an auditory stimulus toward a concurrently-presented visual stimulus. The present study examined whether audiovisual binding, as indicated by the magnitude of the ventriloquist effect, is influenced by threatening auditory stimuli presented prior to the ventriloquist experiment. Syllables spoken in a fearful voice were presented from one of eight loudspeakers while syllables spoken in a neutral voice were presented from the other seven locations. Subsequently, participants had to localize pure tones while trying to ignore concurrent light flashes (both of which were emotionally neutral). A reliable ventriloquist effect was observed. The emotional stimulus manipulation resulted in a reduced ventriloquist effect in both hemifields, as compared to a control group exposed to a similar attention-capturing but non-emotional manipulation. These results suggest that the emotional system is capable of influencing crossmodal binding processes which have heretofore been considered as being automatic.

Speechreading and the Bruce–Young model of face recognition: Early findings and recent developments

In the context of face processing, the skill of processing speech from faces (speechreading) occupies a unique cognitive and neuropsychological niche. Neuropsychological dissociations in two cases (Campbell et al., 1986) suggested a very clear pattern: speechreading, but not face recognition, can be impaired by left-hemisphere damage, while face-recognition impairment consequent to right-hemisphere damage leaves speechreading unaffected. However, this story soon proved too simple, while neuroimaging techniques started to reveal further more detailed patterns. These patterns, moreover, were readily accommodated within the Bruce and Young (1986) model. Speechreading requires structural encoding of faces as faces, but further analysis of visible speech is supported by a network comprising several lateral temporal regions and inferior frontal regions. Posterior superior temporal regions play a significant role in speechreading natural speech, including audiovisual binding in hearing people. In deaf people, similar regions and circuits are implicated. While these detailed developments were not predicted by Bruce and Young, nevertheless, their model has stood the test of time, affording a structural framework for exploring speechreading in terms of face processing.

Searching for audiovisual correspondence in multiple speaker scenarios

A critical question in multisensory processing is how the constant information flow that arrives to our different senses is organized in coherent representations. Some authors claim that pre-attentive detection of inter-sensory correlations supports crossmodal binding, whereas other findings indicate that attention plays a crucial role. We used visual and auditory search tasks for speaking faces to address the role of selective spatial attention in audiovisual binding. Search efficiency amongst faces for the match with a voice declined with the number of faces being monitored concurrently, consistent with an attentive search mechanism. In contrast, search amongst auditory speech streams for the match with a face was independent of the number of streams being monitored concurrently, as long as localization was not required. We suggest that the fundamental differences in the way in which auditory and visual information is encoded play a limiting role in crossmodal binding. Based on these unisensory limitations, we provide a unified explanation for several previous apparently contradictory findings.

EEG-guided fMRI-analysis reveals involvement of the superior temporal sulcus in audio-visual binding

EEG-guided fMRI-analysis reveals involvement of the superior temporal sulcus in audio-visual binding

Visual stimulus locking of EEG is modulated by temporal congruency of auditory stimuli

Disparate sensory streams originating from a common underlying event share similar dynamics, and this plays an important part in multisensory integration. Here we investigate audiovisual binding by presenting continuously changing, temporally congruent and incongruent stimuli. Recorded EEG signals are used to quantify spectrotemporal and waveform locking of neural activity to stimulus dynamics. Spectrotemporal analysis reveals locking to visual stimulus dynamics in both a broad alpha and the beta band. The properties of these effects suggest they are a correlate of bottom-up processing in the visual system. Waveform locking reveals two cortically distinct processes that lock to visual stimulus dynamics with differing topographies and time lags relative to the stimuli. Most importantly, these are modulated in strength by the congruency of an accompanying auditory stream. In addition, the waveform locking found at occipital electrodes shows an increase over stimulus duration for visual and congruent audiovisual stimuli. Hence we argue that these effects reflect audiovisual interaction. We thus propose that spectrotemporal and waveform locking reflect different mechanisms involved in the processing of dynamic audiovisual stimuli.

Multisensory interactions within and outside the focus of visual spatial attention (Commentary on Fairhall & Macaluso)

The environment often engages multiple sensory channels simultaneously. When events or objects stimulate more than one sensory organ, they are more likely to be detected and more efficiently processed than events stimulating just one sensory modality (e.g. Stein & Stanford, 2008). Such crossmodal enhancement requires that the sensory inputs arising from one event are correctly assigned to that object (termed the ‘correspondence problem’; Ernst & Bulthoff, 2004). Multisensory research has demonstrated that features that are shared by all sensory systems, such as space, time or meaning are used for crossmodal binding (for recent reviews see Driver & Noesselt, 2008; Stein & Stanford, 2008). For example, in a cocktail party setting, matching of the temporal dynamics of lip movements and of speech sound allows us to assign the correct message to each speaker. Furthermore, this ability of crossmodal binding assists in orienting spatial attention to a particular conversation partner while ignoring other sources of distracting sensory information, such as other peoples’ conversations. The binding of synchronous auditory and visual (speech) signals seems to happen automatically and pre-attentively (see Driver, 1996; Bertelson et al., 2000; Vroomen et al., 2001a). The voluntary orienting of spatial attention to one particular speaker or, more generally, to an event is an example of endogenous attention allocation. By contrast, stimulus-driven orienting of spatial attention, elicited by the saliency of the stimulus, has been termed exogenous attention. Due to their relatively high salience, such pre-attentively detected, congruent cross-modal stimuli may function as exogenous attentional cues (Spence & Driver, 2000; Stekelenburg et al., 2004; Senkowski et al., 2008). In this issue, Fairhall & Macaluso (2009) provide evidence for the operation of both mechanisms: (i) pre-attentively crossmodal matching and (ii) enhancement of crossmodal interaction by endogenous attention. The authors presented the lips of two speaking faces, one in the left and one in the right visual hemifield. The centrally presented auditory stream was matched with the lip movements of one speaker only. In such a situation people generally perceive the auditory speech signal as originating from the location of the matching lips. This phenomenon is well known as the ‘ventriloquist effect’. Critically, Fairhall and Macaluso also asked participants to perform a visual detection task in one hemifield only. This task served to systematically manipulate the focus of visual spatial attention. The auditory stream did not have any significance to the participants and thus could be ignored. This paradigm allowed the authors to compare brain responses as assessed with functional magnetic resonance imaging (fMRI) to multisensory, matching audio-visual speech sequences that occurred either at the location of visual spatial attention (‘congruent condition’) or on the opposite side (‘incongruent condition’) while keeping physical stimulation constant. The authors report that visual attention enhances activity in brain regions well accepted as multisensory integration areas (Driver & Noesselt, 2008), such as the superior temporal sulcus (STS) and the superior colliculus (SC), only when the auditory track matched the attended lip movements. Similar effects were observed in several visual brain regions, including the primary visual cortex. These results suggest that the processing of congruent cross-modal stimuli within the locus of endogenous visual spatial attention is enhanced. Furthermore, the data of Fairhall & Macaluso are compatible with the view that binding of audio-visual speech can occur pre-attentively. When audio-visual matching speech was presented on the opposite side of the locus of visual-spatial attention (incongruent condition), participants performed worse in the visual detection task than when matching stimuli were presented on the same side as the focus of participant’s endogenous visual spatial attention (congruent condition). These behavioural results, together with findings from earlier studies (Spence & Driver, 2000; Vroomen et al., 2001b), suggest that the audio-visual binding process that elicited a ventriloquist illusion towards the opposite side of the visual attention focus took place outside the locus of endogenous attention and resulted in an exogenously attentional shift towards the task irrelevant side, which in turn interfered with the visual detection task. In summary, the results described by Fairhall and Macaluso suggest that endogenous (visual) spatial attention amplifies the processing of congruent cross-modal input that is likely integrated and detected pre-attentively. Fairhall and Macaluso did not attempt to link the enhanced activity in multisensory and early visual cortical regions to behavioural data such as, for example, improved speech comprehension. The question of whether activity changes in multisensory regions due to endogenous spatial attention are accompanied by altered multisensory performance (Alsius et al., 2005) remains to be investigated in future work. Moreover, other top-down influences on multisensory integration such as plausibility and meaning (Kitagawa & Spence, 2005; Hein et al., 2007) will need to be investigated in more detail to extend our understanding of the conditions when and which multisensory binding processes are subject to voluntary modulation. European Journal of Neuroscience, Vol. 29, pp. 1245–1246, 2009 doi:10.1111/j.1460-9568.2009.06715.x

Effect of lip movement cues on auditory streaming of concurrent speech

Lip reading is known to induce audio-visual integration processes that enhance speech-in-speech intelligibility. Recent work in auditory scene analysis [see Gaudrain et al., Acoustic'08] found correlations between sequential segregation performance and intelligibility of speech-in-speech. Our working hypothesis is that an audiovisual binding process is involved in speech understanding in multi-talker situations. However, the potential effect of audio-visual integration on auditory stream segregation remains unclear. A single previous study [Gaudrain et al., J. Acoust. Soc. Am. 122, 3065 (2007)] found that lip movements congruent with the presentation of vowels may enhance segregation. In this paradigm, audio-visual sequences of vowels with alternating low and high fundamental frequencies have been generated, in which the lip movements are paired with the second ones. The current experiment is built to test further the potential effect of visual cues on segregation. As in the previous study, lips movements were either steady (control condition), open-closed (rhythm condition) or congruent with each particular vowel. Moreover, a congruent but asynchronous audio-visual condition has been introduced as a new control. This is expected to disrupt the audiovisual binding process. Results are interpreted as an evaluation of the contribution of visual cues in Cocktail Party situations.

Attention to touch weakens audiovisual speech integration

One of the classic examples of multisensory integration in humans occurs when speech sounds are combined with the sight of corresponding articulatory gestures. Despite the longstanding assumption that this kind of audiovisual binding operates in an attention-free mode, recent findings (Alsius et al. in Curr Biol, 15(9):839-843, 2005) suggest that audiovisual speech integration decreases when visual or auditory attentional resources are depleted. The present study addressed the generalization of this attention constraint by testing whether a similar decrease in multisensory integration is observed when attention demands are imposed on a sensory domain that is not involved in speech perception, such as touch. We measured the McGurk illusion in a dual task paradigm involving a difficult tactile task. The results showed that the percentage of visually influenced responses to audiovisual stimuli was reduced when attention was diverted to a tactile task. This finding is attributed to a modulatory effect on audiovisual integration of speech mediated by supramodal attention limitations. We suggest that the interactions between the attentional system and crossmodal binding mechanisms may be much more extensive and dynamic than it was advanced in previous studies.

Utilizing the ventriloquism-effect to investigate audio-visual binding

Audio-visual binding – as subset of crossmodal integration – describes the combination of information across both these senses to the subjective unified perception of a bound object. We investigated audio-visual binding by using the ventriloquism-effect (localization of a sound is biased towards and by a simultaneous visual stimulus) to act as an indicator for perceived binding. Simple visual and auditory stimuli were presented synchronously or asynchronously. fMRI was recorded during task performance ( n = 19 subjects) in order to reveal activation in areas discussed to be involved in multisensory processing in the literature. Contrasting trials with reported ventriloquism-effect versus the no-binding condition revealed activation in the insula, superior temporal sulcus and parieto-occipital sulcus. Implementing the ventriloquism-effect allows us to relate these activations to consciousness-related processes, which probably are different from stimulus-driven multisensory integration in subcortical areas.