Audio-visual crossmodal interactions in environmental perception: an fMRI investigation

Abstract

In everyday life, the perception of surrounding events rarely takes place through a single sensory modality. Rather, perception is the result of the processing of information converging from the different senses. Behavioural research often proposes that the binding of different kinds of sensory input creates advantages in the detection, localization, and recognition of external events (King and Calvert 2001). However, the modalities with which multiple sensory cues deriving from the same object merge to form a coherent precept still represent a controversial topic in literature. Research in this field, from the animal studies of Stein and Meredith (1993) to the latest human neuroimaging investigations (for a review see Calvert 2001), has shown the importance of the temporal and spatial congruence of incoming stimuli in establishing crossmodal associations. Nevertheless, other characteristics, like semantic congruence, play a significant role in binding crossmodal associations, especially during the integration of information about complex objects. Thus the analysis of the semantic relationships established in crossmodal stimulation may be a promising instrument for exploring the neural substrates of multi-sensory integration. At least two main theoretical views on the neural pathways involved in crossmodal processing have been proposed: the first one stresses the importance of multisensory cortical areas that receive projections from the different senses (Calvert 2001); the second view emphasizes the importance of the combined activity of the modality-specific cortices by means of synchronized firing (Ettlinger and Wilson 1990). At the same time, an effort has been made to identify regions of the human brain, called heteromodal cortices, that receive afferents from different senses and are analogous to those described in animals. At the cortical level, these regions have been found in the superior temporal sulcus, in the intraparietal sulcus, and in the prefrontal and limbic cortices (Mesulam 1998; Calvert 2001). Heteromodal areas have also been found in subcortical structures, such as the superior colliculus (Stein and Meredith 1993). Recent neuroimaging investigations attempted to establish a relationship between different crossmodal tasks and the activation of specific heteromodal areas (Calvert 2001). Experimental evidence reported the activation of the lateral temporal cortex in response to the integration of audio-visual information during recognition or identification tasks, as described by Calvert et al. (2000). These authors investigated the regions that exhibited supraadditive response enhancement to congruent audiovisual speech using functional magnetic resonance imaging (fMRI). They observed strong interaction effects in the left superior temporal sulcus. Recently, Beauchamp et al. (2004) investigated the contribution of the superior temporal areas in the integration of visual and auditory information about Cogn Process (2004) 5: 167–174 DOI 10.1007/s10339-004-0024-0