Human Multisensory Integration Research Articles

Multisensory integration in humans with spinal cord injury

Although multisensory integration (MSI) has been extensively studied, the underlying mechanisms remain a topic of ongoing debate. Here we investigate these mechanisms by comparing MSI in healthy controls to a clinical population with spinal cord injury (SCI). Deafferentation following SCI induces sensorimotor impairment, which may alter the ability to synthesize cross-modal information. We applied mathematical and computational modeling to reaction time data recorded in response to temporally congruent cross-modal stimuli. We found that MSI in both SCI and healthy controls is best explained by cross-modal perceptual competition, highlighting a common competition mechanism. Relative to controls, MSI impairments in SCI participants were better explained by reduced stimulus salience leading to increased cross-modal competition. By combining traditional analyses with model-based approaches, we examine how MSI is realized during normal function, and how it is compromised in a clinical population. Our findings support future investigations identifying and rehabilitating MSI deficits in clinical disorders.

Virtual reality alters cortical oscillations related to visuo-tactile integration during rubber hand illusion

Virtual reality (VR) enables the fast, free, and highly controllable setting of experimental body images. Illusions pertaining to a body, such as the rubber hand illusion (RHI), can be easily conducted in VR settings, and some phenomena, such as full-body illusions, are only realized in virtual environments. However, the multisensory-integration process in VR is not yet fully understood. Thus, it remains to be clarified if specific phenomena that occur under VR settings manifest in real life as well. One useful investigative approach is measuring brain activities during a psychological experiment. Electroencephalography (EEG) oscillatory activities provide insight into the human multisensory integration process. Nevertheless, EEG data can be vulnerable to VR noise, which causes measurement and analytical difficulties for EEG data recorded in VR environments. Here, we achieve an experimental RHI setting using a head-mounted display that provides a VR visual space and VR dummy hand along with EEG measurements. We compared EEG data collected in both real and VR environments and observed the gamma and theta band oscillatory activities. Ultimately, we observed statistically significant differences between congruent (RHI) and incongruent (not RHI) conditions in the real environment, which is consistent with previous studies. Differences in the VR condition were observed only on the late theta band oscillation, suggesting that the VR setting itself altered the perceptual and sensory integration mechanisms. Thus, we must model this difference between real and VR settings whenever we use VR to investigate our bodily self-perception.

Approaching the Ground Truth: Revealing the Functional Organization of Human Multisensory STC Using Ultra-High Field fMRI.

Integrating inputs across sensory systems is a property of the brain that is vitally important in everyday life. More than two decades of fMRI research have revealed crucial insights on multisensory processing, yet the multisensory operations at the neuronal level in humans have remained largely unknown. Understanding the fine-scale spatial organization of multisensory brain regions is fundamental to shed light on their neuronal operations. Monkey electrophysiology revealed that the bimodal superior temporal cortex (bSTC) is topographically organized according to the modality preference (visual, auditory, and bimodal) of its neurons. In line with invasive studies, a previous 3 Tesla fMRI study suggests that the human bSTC is also topographically organized according to modality preference (visual, auditory, and bimodal) when analyzed at 1.6 × 1.6 × 1.6 mm3 voxel resolution. However, it is still unclear whether this resolution is able to unveil an accurate spatial organization of the human bSTC. This issue was addressed in the present study by investigating the spatial organization of functional responses of the bSTC in 10 participants (from both sexes) at 1.5 × 1.5 × 1.5 mm3 and 1.1 × 1.1 × 1.1 mm3 using ultra-high field fMRI (at 7 Tesla). Relative to 1.5 × 1.5 × 1.5 mm3, the bSTC at 1.1 × 1.1 × 1.1 mm3 resolution was characterized by a larger selectivity for visual and auditory modalities, stronger integrative responses in bimodal voxels, and it was organized in more distinct functional clusters indicating a more precise separation of underlying neuronal clusters. Our findings indicate that increasing the spatial resolution may be necessary and sufficient to achieve a more accurate functional topography of human multisensory integration.SIGNIFICANCE STATEMENT The bimodal superior temporal cortex (bSTC) is a brain region that plays a crucial role in the integration of visual and auditory inputs. The aim of the present study was to investigate the fine-scale spatial organization of the bSTC by using ultra-high magnetic field fMRI at 7 Tesla. Mapping the functional topography of bSTC at a resolution of 1.1 × 1.1 × 1.1 mm3 revealed more accurate representations than at lower resolutions. This result indicates that standard-resolution fMRI may lead to wrong conclusions about the functional organization of the bSTC, whereas high spatial resolution is essential to more accurately approach neuronal operations of human multisensory integration.

Modeling development of natural multi-sensory integration using neural self-organisation and probabilistic population codes

Humans and other animals have been shown to perform near-optimally in multi-sensory integration tasks. Probabilistic population codes (PPCs) have been proposed as a mechanism by which optimal integration can be accomplished. Previous approaches have focussed on how neural networks might produce PPCs from sensory input or perform calculations using them, like combining multiple PPCs. Less attention has been given to the question of how the necessary organisation of neurons can arise and how the required knowledge about the input statistics can be learned. In this paper, we propose a model of learning multi-sensory integration based on an unsupervised learning algorithm in which an artificial neural network learns the noise characteristics of each of its sources of input. Our algorithm borrows from the self-organising map the ability to learn latent-variable models of the input and extends it to learning to produce a PPC approximating a probability density function over the latent variable behind its (noisy) input. The neurons in our network are only required to perform simple calculations and we make few assumptions about input noise properties and tuning functions. We report on a neurorobotic experiment in which we apply our algorithm to multi-sensory integration in a humanoid robot to demonstrate its effectiveness and compare it to human multi-sensory integration on the behavioural level. We also show in simulations that our algorithm performs near-optimally under certain plausible conditions, and that it reproduces important aspects of natural multi-sensory integration on the neural level.

Just how important is spatial coincidence to multisensory integration? Evaluating the spatial rule

This review addresses the question of when spatial coincidence facilitates multisensory integration in humans. According to the spatial rule (which was first formulated on the basis of neurophysiological data in anesthetized animals), multisensory integration is enhanced when stimuli in different sensory modalities are presented from the same spatial location. While the spatial rule fits with the available data from studies of overt and covert spatial attentional orienting, and from the majority of those studies in which space has been somehow relevant to the participant's task, it is inconsistent with the evidence that has emerged from the majority of multisensory studies of stimulus identification and temporal perception. Such a mixed pattern of behavioral results suggests that the spatial rule does not represent a general constraint on multisensory integration in humans. Instead, it would appear to be a much more task-dependent phenomenon than is often realized. These results, however, are broadly consistent with a distinction between the processing of "where" and "what" (or "how") information processing in the human brain.

Multisensory visual–tactile object related network in humans: insights gained using a novel crossmodal adaptation approach

Neuroimaging techniques have provided ample evidence for multisensory integration in humans. However, it is not clear whether this integration occurs at the neuronal level or whether it reflects areal convergence without such integration. To examine this issue as regards visuo-tactile object integration we used the repetition suppression effect, also known as the fMRI-based adaptation paradigm (fMR-A). Under some assumptions, fMR-A can tag specific neuronal populations within an area and investigate their characteristics. This technique has been used extensively in unisensory studies. Here we applied it for the first time to study multisensory integration and identified a network of occipital (LOtv and calcarine sulcus), parietal (aIPS), and prefrontal (precentral sulcus and the insula) areas all showing a clear crossmodal repetition suppression effect. These results provide a crucial first insight into the neuronal basis of visuo-haptic integration of objects in humans and highlight the power of using fMR-A to study multisensory integration using non-invasinve neuroimaging techniques.Electronic supplementary materialThe online version of this article (doi:10.1007/s00221-009-1949-4) contains supplementary material, which is available to authorized users.

Assessing the role of attention in the audiovisual integration of speech

Currently, one of the most controversial topics in the study of multisensory integration in humans (and in its implementation in the development of new technologies for human communication systems) concerns the question of whether or not attention is needed during (or can modulate) the integration of sensory signals that are presented in different sensory modalities. Here, we review the evidence on this question, focusing specifically on the integration of auditory and visual information during the perception of speech. Contrary to the mainstream view that has been prevalent for the last 30 years or so, recent studies have now started to reveal that attentional resources are, in fact, recruited during audiovisual multisensory integration, at least under certain conditions. Finally, considering all of the available evidence, we discuss the extent to which audiovisual speech perception should be considered to represent a ‘special’ case of audiovisual, and more generally, of multisensory integration.

Multisensory Integration

Multisensory integration in humans is thought to be essentially a brain phenomenon, but theories are silent as to the possible involvement of the peripheral nervous system. We provide evidence that this approach is insufficient. We report novel tactile-auditory and tactile-visual interactions in humans, demonstrating that a facilitating sound or visual stimulus that is exactly synchronous with an excitatory tactile signal presented at the lower leg increases the peripheral representation of that excitatory signal. These results demonstrate that during multisensory integration, the brain not only continuously binds information obtained from the senses, but also acts directly on that information by modulating activity at peripheral levels. We also discuss a theoretical framework to explain this novel interaction.

Attention to touch weakens audiovisual speech integration

One of the classic examples of multisensory integration in humans occurs when speech sounds are combined with the sight of corresponding articulatory gestures. Despite the longstanding assumption that this kind of audiovisual binding operates in an attention-free mode, recent findings (Alsius et al. in Curr Biol, 15(9):839-843, 2005) suggest that audiovisual speech integration decreases when visual or auditory attentional resources are depleted. The present study addressed the generalization of this attention constraint by testing whether a similar decrease in multisensory integration is observed when attention demands are imposed on a sensory domain that is not involved in speech perception, such as touch. We measured the McGurk illusion in a dual task paradigm involving a difficult tactile task. The results showed that the percentage of visually influenced responses to audiovisual stimuli was reduced when attention was diverted to a tactile task. This finding is attributed to a modulatory effect on audiovisual integration of speech mediated by supramodal attention limitations. We suggest that the interactions between the attentional system and crossmodal binding mechanisms may be much more extensive and dynamic than it was advanced in previous studies.

Audiovisual Integration of Speech Falters under High Attention Demands

One of the most commonly cited examples of human multisensory integration occurs during exposure to natural speech, when the vocal and the visual aspects of the signal are integrated in a unitary percept. Audiovisual association of facial gestures and vocal sounds has been demonstrated in nonhuman primates and in prelinguistic children, arguing for a general basis for this capacity. One critical question, however, concerns the role of attention in such multisensory integration. Although both behavioral and neurophysiological studies have converged on a preattentive conceptualization of audiovisual speech integration, this mechanism has rarely been measured under conditions of high attentional load, when the observers' attention resources are depleted. We tested the extent to which audiovisual integration was modulated by the amount of available attentional resources by measuring the observers' susceptibility to the classic McGurk illusion in a dual-task paradigm. The proportion of visually influenced responses was severely, and selectively, reduced if participants were concurrently performing an unrelated visual or auditory task. In contrast with the assumption that crossmodal speech integration is automatic, our results suggest that these multisensory binding processes are subject to attentional demands.

Recent results in emergent visual segmentation

One of the most commonly cited examples of human multisensory integration occurs during exposure to natural speech, when the vocal and the visual aspects of the signal are integrated in a unitary percept. Audiovisual association of facial gestures and vocal sounds has been demonstrated in nonhuman primates [1] and in prelinguistic children [2], arguing for a general basis for this capacity. One critical question, however, concerns the role of attention in such multisensory integration. Although both behavioral and neurophysiological studies have converged on a preattentive conceptualization of audiovisual speech integration [3, 4, 5, 6, 7, 8], this mechanism has rarely been measured under conditions of high attentional load, when the observers’ attention resources are depleted [9]. We tested the extent to which audiovisual integration was modulated by the amount of available attentional resources by measuring the observers’ susceptibility to the classic McGurk illusion [3] in a dual-task paradigm [10]. The proportion of visually influenced responses was severely, and selectively, reduced if participants were concurrently performing an unrelated visual or auditory task. In contrast with the assumption that crossmodal speech integration is automatic, our results suggest that these multisensory binding processes are subject to attentional demands.