Multisensory Effects Research Articles

Multisensory cueing and the attention network test in aging

The Attention Network Test (ANT) assesses the effect of alerting and orienting cues on a visual flanker task measuring executive attention. Previous findings revealed that older adults demonstrate greater RT benefits when provided with visual orienting cues that offer both spatial and temporal information of an ensuing target. Given the overlap of neural correlates involved in multisensory processing and cueing (i.e., alerting and orienting), especially in the superior colliculus, thalamus, superior temporal and parietal regions, an investigation of multisensory cueing effects was warranted. The current study was designed to determine whether participants, both old and young, benefited from receiving multisensory alerting and orienting cues on a visual flanker task. Eighteen young (M = 19.17 yrs) and eighteen old (M = 76.44 yrs) individuals that were determined to be non-demented and without any medical or psychiatric conditions that would affect their performance were included. Results revealed main effects for the executive attention and orienting networks, but not for the alerting network. In terms of orienting, both old and young adults demonstrated significant orienting effects for auditory–somatosensory (AS), auditory–visual (AV), and visual–somatosensory (VS) cues. Benefits of multisensory compared to unisensory averaged orienting effects differed by cue type and age group; younger adults demonstrated significantly greater RT benefits for AS orienting cues whereas older adults demonstrated significantly greater RT benefits for AV orienting cues. Both groups, however, demonstrated significant RT benefits for VS orienting cues. These findings provide evidence for the facilitative effect of multisensory orienting cues, and not multisensory alerting cues, in old and young adults.

Auditory modulation of oscillatory activity in extra-striate visual cortex and its contribution to audio–visual multisensory integration: A human intracranial EEG study

Investigations have traditionally focused on activity in the sensory cortices as a function of their respective sensory inputs. However, converging evidence from multisensory research has shown that neural activity in a given sensory region can be modulated by stimulation of other so-called ancillary sensory systems. Both electrophysiology and functional imaging support the occurrence of multisensory processing in human sensory cortex based on the latency of multisensory effects and their precise anatomical localization. Still, due to inherent methodological limitations, direct evidence of the precise mechanisms by which multisensory integration occurs within human sensory cortices is lacking. Using intracranial recordings in epileptic patients () undergoing presurgical evaluation, we investigated the neurophysiological basis of multisensory integration in visual cortex. Subdural electrical brain activity was recorded while patients performed a simple detection task of randomly ordered Auditory alone (A), Visual alone (V) and Audio–Visual stimuli (AV). We then performed time-frequency analysis: first we investigated each condition separately to evaluate responses compared to baseline, then we indexed multisensory integration using both the maximum criterion model (AV vs. V) and the additive model (AV vs. A+V). Our results show that auditory input significantly modulates neuronal activity in visual cortex by resetting the phase of ongoing oscillatory activity. This in turn leads to multisensory integration when auditory and visual stimuli are simultaneously presented.

Integration of faces and voices, but not faces and names, in person recognition

Recent studies on cross-modal recognition suggest that face and voice information are linked for the purpose of person identification. We tested whether congruent associations between familiarized faces and voices facilitated subsequent person recognition relative to incongruent associations. Furthermore, we investigated whether congruent face and name associations would similarly benefit person identification relative to incongruent face and name associations. Participants were familiarized with a set of talking video-images of actors, their names, and their voices. They were then tested on their recognition of either the face, voice, or name of each actor from bimodal stimuli which were either congruent or novel (incongruent) associations between the familiarized face and voice or face and name. We found that response times to familiarity decisions based on congruent face and voice stimuli were facilitated relative to incongruent associations. In contrast, we failed to find a benefit for congruent face and name pairs. Our findings suggest that faces and voices, but not faces and names, are integrated in memory for the purpose of person recognition. These findings have important implications for current models of face perception and support growing evidence for multisensory effects in face perception areas of the brain for the purpose of person recognition.

Multisensory Effects Differ for Counting Small and Large Pulse Numbers

In Illusory Flash (IF) experiments, congruent multisensory presentation has no effect on the mean estimate of the number of events, but decreases the variance in comparison with unisensory presentation. In contrast, congruent multisensory presentation in other Temporal Numerosity Judgement (TNJ) tasks affects the mean estimate (i.e., it often results in a reduction in underestimation) and increases the variance. In three experiments, we investigated the differences between both paradigms as possible causes of this discrepancy: the presence or absence of incongruent stimuli (Experiment 1), the instruction to the observer to either count flashes, beeps or multisensory events (Experiment 2), and the range of pulses presented (Experiment 3). We found significant differences between the mean numerosity estimate of multisensory and unisensory series in Experiment 3, but not in 1 and 2. This suggests that the difference in the range of pulses presented in IF (1-3 pulses) and TNJ (1-10 pulses) is the primary cause of the discrepancy. In the discussion we propose that this result may be explained by the use of two different strategies and their susceptibility to multisensory presentation. For small pulse numbers, observers can accurately count both unisensory and multisensory pulses. For larger numbers, observers can no longer count but will estimate the number based on the pulse series duration which is improved for multisensory stimuli.

Projections to Early Visual Areas V1 and V2 in the Calcarine Fissure from Parietal Association Areas in the Macaque

Non-extrastriate projections to area V1 in monkeys, now demonstrated by several anatomical studies, are potential substrates of physiologically documented multisensory effects in primary sensory areas. The full network of projections among association and primary areas, however, is likely to be complex and is still only partially understood. In the present report, we used the anterograde tracer biotinylated dextran amine to investigate projections to areas V1 and V2 from subdivisions of the parietal association cortex in macaque. Parietal cortex was chosen to allow comparisons between projections from this higher association area and from other previously reported areas. In addition, we were interested in further elucidating pathways to areas V1 and V2 from parietal areas, as potentially contributing to attention and active vision. Of eight cases, three brains had projections only to area V2, and the five others projected to both areas V1 and V2. Terminations in area V1 were sparse. These were located in supragranular layers I, II, upper III; occasionally in IVB; and in layer VI. Terminations in V2 were denser, and slightly more prevalent in the supragranular layers. For both areas, terminations were in the calcarine region, corresponding to the representation of the peripheral visual field. By reconstructions of single axons, we demonstrated that four of nine axons had collaterals, either to V1 and V2 (n = 1) or to area V1 and a ventral area likely to be TEO (n = 3). In area V1, axons extended divergently in layer VI as well as layer I. Overall, these and previous results suggest a nested connectivity architecture, consisting of multiple direct and indirect recurrent projections from association areas to area V1. Terminations in area V1 are not abundant, but could be potentiated by the network of indirect connections.

The representation of space near the body through touch and vision

This review discusses how visual and the tactile signals are combined in the brain to ensure appropriate interactions with the space around the body. Visual and tactile signals converge in many regions of the brain (e.g. parietal and premotor cortices) where multisensory input can interact on the basis of specific spatial constraints. Crossmodal interactions can modulate also unisensory visual and somatosensory cortices, possibly via feed-back projections from fronto-parietal areas. These processes enable attentional selection of relevant locations in near body space, as demonstrated by studies of spatial attention in healthy volunteers and in neuropsychological patients with crossmodal extinction. These crossmodal spatial effects can be flexibly updated taking into account the position of the eyes and the limbs, thus reflecting the spatial alignment of visuo-tactile stimuli in external space. Further, studies that manipulated vision of body parts (alien, real or fake limbs) have demonstrated that passive viewing of the body can influence the perception of somatosensory stimuli, again involving areas in the premotor and parietal cortices. Finally, we discuss how tool-use can expand the region of visuo-tactile integration in near body space, emphasizing the flexibility of this system at the single-neuron level in the monkey's parietal cortex, with corresponding multisensory effects in normals and neuropsychological patients. We conclude that visuo-tactile crossmodal links dominate the representation of near body space and that this is implemented functionally in parietal and premotor brain regions. These integration processes mediate the orienting of spatial attention and generate an efficient and flexible representation the space around the body.

Not Just for Bimodal Neurons Anymore: The Contribution of Unimodal Neurons to Cortical Multisensory Processing

Traditionally, neuronal studies of multisensory processing proceeded by first identifying neurons that were overtly multisensory (e.g., bimodal, trimodal) and then testing them. In contrast, the present study examined, without precondition, neurons in an extrastriate visual area of the cat for their responses to separate (visual, auditory) and combined-modality (visual and auditory) stimulation. As expected, traditional bimodal forms of multisensory neurons were identified. In addition, however, many neurons that were activated only by visual stimulation (i.e., unimodal) had that response modulated by the presence of an auditory stimulus. Some unimodal neurons showed multisensory responses that were statistically different from their visual response. Other unimodal neurons had subtle multisensory effects that were detectable only at the population level. Most surprisingly, these non-bimodal neurons generated more than twice the multisensory signal in the PLLS than did the bimodal neurons. These results expand the range of multisensory convergence patterns beyond that of the bimodal neuron. However, rather than characterize a separate class of multisensory neurons, unimodal multisensory neurons may actually represent an intermediary form of multisensory convergence that exists along the functional continuum between unisensory neurons, at one end, and fully bimodal neurons at the other.

Provision of the expressive multisensory adaptation platform for heterogeneous multimedia devices in the ubiquitous home

This paper presents a multimedia service platform to express multisensory media effects for heterogeneous appliances and a home server in the ubiquitous home networks. The platform is equipped with an adaptation technology to process multimedia service based on a home server. The proposed adaptation consists of a multimedia parser, analyzer, device profile gatherer, device selector and a media player. The multisensory adaptation communicates with service modules for integrating the control of various home appliances. The multimedia service supports not only multimedia contents subscriptions but also presents additional sensory effects to the high quality multimedia services for various different homes via the adaptation.

Early, low-level auditory-somatosensory multisensory interactions impact reaction time speed

Several lines of research have documented early-latency non-linear response interactions between audition and touch in humans and non-human primates. That these effects have been obtained under anesthesia, passive stimulation, as well as speeded reaction time tasks would suggest that some multisensory effects are not directly influencing behavioral outcome. We investigated whether the initial non-linear neural response interactions have a direct bearing on the speed of reaction times. Electrical neuroimaging analyses were applied to event-related potentials in response to auditory, somatosensory, or simultaneous auditory–somatosensory multisensory stimulation that were in turn averaged according to trials leading to fast and slow reaction times (using a median split of individual subject data for each experimental condition). Responses to multisensory stimulus pairs were contrasted with each unisensory response as well as summed responses from the constituent unisensory conditions. Behavioral analyses indicated that neural response interactions were only implicated in the case of trials producing fast reaction times, as evidenced by facilitation in excess of probability summation. In agreement, supra-additive non-linear neural response interactions between multisensory and the sum of the constituent unisensory stimuli were evident over the 40–84 ms post-stimulus period only when reaction times were fast, whereas subsequent effects (86–128 ms) were observed independently of reaction time speed. Distributed source estimations further revealed that these earlier effects followed from supra-additive modulation of activity within posterior superior temporal cortices. These results indicate the behavioral relevance of early multisensory phenomena.

Do Cross-Modal Projections Always Result in Multisensory Integration?

Convergence of afferents from different sensory modalities has generally been thought to produce bimodal (and trimodal) neurons (i.e., exhibit suprathreshold excitation to more than 1 sensory modality). Consequently, studies identifying cross-modal connections assume that such convergence results in bimodal (or trimodal) neurons that produce familiar forms of multisensory integration: response enhancement or depression. The present study questioned that assumption by anatomically identifying a projection from ferret auditory to visual cortex Area 21. However, electrophysiological recording within Area 21 not only failed to identify a single bimodal neuron but also familiar forms of multisensory integration were not observed either. Instead, a small proportion of neurons (9%; 27/296) showed subthreshold multisensory integration, in which visual responses were significantly modulated by auditory inputs. Such subthreshold multisensory effects were enhanced by gamma-aminobutyric acid antagonism, whereby a majority of neurons (87%; 20/23) now participated in a significant, multisensory population effect. Thus, multisensory convergence does not de facto result in bimodal (or trimodal) neurons or the traditional forms of multisensory integration. However, the fact that unimodal neurons exhibited a subthreshold form of multisensory integration not only affirms the relationship between convergence and integration but also expands our understanding of the functional repertoire of multisensory processing itself.

Tool use changes multisensory interactions in seconds: evidence from the crossmodal congruency task

Active tool use in human and non-human primates has been claimed to alter the neural representations of multisensory peripersonal space. To date, most studies suggest that a short period of tool use leads to an expansion or elongation of these spatial representations, which lasts several minutes after the last tool use action. However, the possibility that multisensory interactions also change on a much shorter time scale following or preceding individual tool use movements has not yet been investigated. We measured crossmodal (visual-tactile) congruency effects as an index of multisensory integration during two tool use tasks. In the regular tool use task, the participants used one of two tools in a spatiotemporally predictable sequence after every fourth crossmodal congruency trial. In the random tool use task, the required timing and spatial location of the tool use task varied unpredictably. Multisensory integration effects increased as a function of the number of trials since tool use in the regular tool use group, but remained relatively constant in the random tool use group. The spatial distribution of these multisensory effects, however, was unaffected by tool use predictability, with significant spatial interactions found only near the hands and at the tips of the tools. These data suggest that endogenously preparing to use a tool enhances visual-tactile interactions near the tools. Such enhancements are likely due to the increased behavioural relevance of visual stimuli as each tool use action is prepared before execution.

Crossmodal projections from somatosensory area SIV to the auditory field of the anterior ectosylvian sulcus (FAES) in Cat: further evidence for subthreshold forms of multisensory processing

To date, evaluation of the neuronal basis for multisensory processing has focused on the convergence pattern that provides excitation from more than one sensory modality. However, a recent study (Dehner et al. in Cereb Cortex 14:387-401, 2004) has demonstrated excitatory-inhibitory multisensory effects that do not follow this conventional pattern and the present investigation documented a similar example of subthreshold cross-modal effects. Neuroanatomical tracers revealed that pyramidal neurons of the somatosensory area SIV project to the auditory field of the anterior ectosylvian sulcus (FAES), but subsequent electrophysiological tests showed that stimulation of SIV failed to elicit the expected orthodromic responses in FAES. Instead, combined auditory-SIV stimulation significantly suppressed FAES responses to auditory cues in approximately 25% of the neurons tested, and facilitated responses in another 5%. These modulatory responses in auditory FAES were similar in kind to those observed in somatosensory SIV and, as such, comprise further evidence for subthreshold forms of multisensory processing in cortex. Consequently, it seems likely that subthreshold cross-modal effects may impact other apparently 'unimodal' areas of the brain.

Multisensory visual-auditory object recognition in humans: a high-density electrical mapping study.

Multisensory object-recognition processes were investigated by examining the combined influence of visual and auditory inputs upon object identification--in this case, pictures and vocalizations of animals. Behaviorally, subjects were significantly faster and more accurate at identifying targets when the picture and vocalization were matched (i.e. from the same animal), than when the target was represented in only one sensory modality. This behavioral enhancement was accompanied by a modulation of the evoked potential in the latency range and general topographic region of the visual evoked N1 component, which is associated with early feature processing in the ventral visual stream. High-density topographic mapping and dipole modeling of this multisensory effect were consistent with generators in lateral occipito-temporal cortices, suggesting that auditory inputs were modulating processing in regions of the lateral occipital cortices. Both the timing and scalp topography of this modulation suggests that there are multisensory effects during what is considered to be a relatively early stage of visual object-recognition processes, and that this modulation occurs in regions of the visual system that have traditionally been held to be unisensory processing areas. Multisensory inputs also modulated the visual 'selection-negativity', an attention dependent component of the evoked potential this is usually evoked when subjects selectively attend to a particular feature of a visual stimulus.

Different aspects of the face are represented in the dynamic firing rate patterns of macaque temporal cortical neurons

This review discusses how visual and the tactile signals are combined in the brain to ensure appropriate interactions with the space around the body. Visual and tactile signals converge in many regions of the brain (e.g. parietal and premotor cortices) where multisensory input can interact on the basis of specific spatial constraints. Crossmodal interactions can modulate also unisensory visual and somatosensory cortices, possibly via feed-back projections from fronto-parietal areas. These processes enable attentional selection of relevant locations in near body space, as demonstrated by studies of spatial attention in healthy volunteers and in neuropsychological patients with crossmodal extinction. These crossmodal spatial effects can be flexibly updated taking into account the position of the eyes and the limbs, thus reflecting the spatial alignment of visuo-tactile stimuli in external space. Further, studies that manipulated vision of body parts (alien, real or fake limbs) have demonstrated that passive viewing of the body can influence the perception of somatosensory stimuli, again involving areas in the premotor and parietal cortices. Finally, we discuss how tool-use can expand the region of visuo-tactile integration in near body space, emphasizing the flexibility of this system at the single-neuron level in the monkey's parietal cortex, with corresponding multisensory effects in normals and neuropsychological patients. We conclude that visuo-tactile crossmodal links dominate the representation of near body space and that this is implemented functionally in parietal and premotor brain regions. These integration processes mediate the orienting of spatial attention and generate an efficient and flexible representation the space around the body.