Multisensory Interplay Research Articles

Cross-modal perceptual enhancement of unisensory targets is uni-directional and does not affect temporal expectations

Temporal structures in the environment can shape temporal expectations (TE); and previous studies demonstrated that TEs interact with multisensory interplay (MSI) when multisensory stimuli are presented synchronously. Here, we tested whether other types of MSI – evoked by asynchronous yet temporally flanking irrelevant stimuli – result in similar performance patterns. To this end, we presented sequences of 12 stimuli (10 Hz) which consisted of auditory (A), visual (V) or alternating auditory-visual stimuli (e.g. A-V-A-V-…) with either auditory or visual targets (Exp. 1). Participants discriminated target frequencies (auditory pitch or visual spatial frequency) embedded in these sequences. To test effects of TE, the proportion of early and late temporal target positions was manipulated run-wise. Performance for unisensory targets was affected by temporally flanking distractors, with auditory temporal flankers selectively improving visual target perception (Exp. 1). However, no effect of temporal expectation was observed. Control experiments (Exp. 2–3) tested whether this lack of TE effect was due to the higher presentation frequency in Exp. 1 relative to previous experiments. Importantly, even at higher stimulation frequencies redundant multisensory targets (Exp. 2–3) reliably modulated TEs. Together, our results indicate that visual target detection was enhanced by MSI. However, this cross-modal enhancement – in contrast to the redundant target effect – was still insufficient to generate TEs. We posit that unisensory target representations were either instable or insufficient for the generation of TEs while less demanding MSI still occurred; highlighting the need for robust stimulus representations when generating temporal expectations.

Synchronisation of Neural Oscillations and Cross-modal Influences.

At any given moment, we receive multiple signals from our different senses. Prior research has shown that signals in one sensory modality can influence neural activity and behavioural performance associated with another sensory modality. Recent human and nonhuman primate studies suggest that such cross-modal influences in sensory cortices are mediated by the synchronisation of ongoing neural oscillations. In this review, we consider two mechanisms proposed to facilitate cross-modal influences on sensory processing, namely cross-modal phase resetting and neural entrainment. We consider how top-down processes may further influence cross-modal processing in a flexible manner, and we highlight fruitful directions for further research.

Vestibular-guided visual search

The amnesic symptoms that accompany vestibular dysfunction point to a functional relationship between the vestibular and visual memory systems. However, little is known about the underpinning cognitive processes. As a starting point, we sought evidence for a type of cross-modal interaction commonly observed between other sensory modalities in which the identification of a target (in this case, visual) is facilitated if earlier coupled to a unique, temporally coincident stimulus from another sensory domain (in this case, vestibular). Participants first performed a visual detection task in which stimuli appeared at random locations within a computerised grid. Unknown to participants, the onset of one particular stimulus was accompanied by a brief, sub-sensory pulse of galvanic vestibular stimulation (GVS). Across two visual search experiments, both old and new targets were identified faster when presented in the grid location at which the GVS-paired visual stimulus had appeared in the earlier detection task. This location advantage appeared to be based on relative rather than absolute spatial co-ordinates since the effect held when the search grid was rotated 90°. Together these findings indicate that when individuals return to a familiar visual scene (here, a 2D grid), visual judgements are facilitated when targets appear at a location previously associated with a unique, task-irrelevant vestibular cue. This novel case of multisensory interplay has broader implications for understanding how vestibular signals inform cognitive processes and helps constrain the growing therapeutic application of GVS.

Phasic and sustained interactions of multisensory interplay and temporal expectation

Every moment organisms are confronted with complex streams of information which they use to generate a reliable mental model of the world. There is converging evidence for several optimization mechanisms instrumental in integrating (or segregating) incoming information; among them are multisensory interplay (MSI) and temporal expectation (TE). Both mechanisms can account for enhanced perceptual sensitivity and are well studied in isolation; how these two mechanisms interact is currently less well-known. Here, we tested in a series of four psychophysical experiments for TE effects in uni- and multisensory contexts with different levels of modality-related and spatial uncertainty. We found that TE enhanced perceptual sensitivity for the multisensory relative to the best unisensory condition (i.e. multisensory facilitation according to the max-criterion). In the latter TE effects even vanished if stimulus-related spatial uncertainty was increased. Accordingly, computational modelling indicated that TE, modality-related and spatial uncertainty predict multisensory facilitation. Finally, the analysis of stimulus history revealed that matching expectation at trial n-1 selectively improves multisensory performance irrespective of stimulus-related uncertainty. Together, our results indicate that benefits of multisensory stimulation are enhanced by TE especially in noisy environments, which allows for more robust information extraction to boost performance on both short and sustained time ranges.

The role of multisensory interplay in enabling temporal expectations

Temporal regularities can guide our attention to focus on a particular moment in time and to be especially vigilant just then. Previous research provided evidence for the influence of temporal expectation on perceptual processing in unisensory auditory, visual, and tactile contexts. However, in real life we are often exposed to a complex and continuous stream of multisensory events. Here we tested – in a series of experiments – whether temporal expectations can enhance perception in multisensory contexts and whether this enhancement differs from enhancements in unisensory contexts. Our discrimination paradigm contained near-threshold targets (subject-specific 75% discrimination accuracy) embedded in a sequence of distractors. The likelihood of target occurrence (early or late) was manipulated block-wise. Furthermore, we tested whether spatial and modality-specific target uncertainty (i.e. predictable vs. unpredictable target position or modality) would affect temporal expectation (TE) measured with perceptual sensitivity (d′) and response times (RT). In all our experiments, hidden temporal regularities improved performance for expected multisensory targets. Moreover, multisensory performance was unaffected by spatial and modality-specific uncertainty, whereas unisensory TE effects on d′ but not RT were modulated by spatial and modality-specific uncertainty. Additionally, the size of the temporal expectation effect, i.e. the increase in perceptual sensitivity and decrease of RT, scaled linearly with the likelihood of expected targets. Finally, temporal expectation effects were unaffected by varying target position within the stream. Together, our results strongly suggest that participants quickly adapt to novel temporal contexts, that they benefit from multisensory (relative to unisensory) stimulation and that multisensory benefits are maximal if the stimulus-driven uncertainty is highest. We propose that enhanced informational content (i.e. multisensory stimulation) enables the robust extraction of temporal regularities which in turn boost (uni-)sensory representations.

Comment from the Editor

The primary auditory cortex, which was previously considered to be unisensory, has been shown to be multisensory. However, the temporal details of processing nonauditory stimuli in the human auditory cortex remain unclear, owing to the low temporal and spatial resolution of the adopted imaging techniques.Using intraoperative optical imaging of intrinsic signals recording techniques, detailed cortical activations within the auditory cortex in response to auditory and somatosensory stimulation were recorded from 3 intraoperative anesthetized patients with brain tumors located in the superior temporal gyrus.At both green-light (545 nm ± 13) and red-light (610 nm ± 10) illumination, the primary and secondary auditory cortices responded significantly to the somatosensory stimulation. As induced by the somatosensory stimulus, the average overlapping rate of the activated region was 74.51% ± 0.15, and the peak responding time occurred at poststimulus 7–8 seconds. There was no significant difference in the peak responding time between auditory and somatosensory stimuli (P < 0.01, paired t test).These findings provide novel evidence for multisensory interplay within human auditory cortex at an early stage of cortical processing, which extends the understanding of multisensory mechanisms of human brain functions.

Multisensory Interplay Within Human Auditory Cortex: New Evidence from Intraoperative Optical Imaging of Intrinsic Signal

The primary auditory cortex, which was previously considered to be unisensory, has been shown to be multisensory. However, the temporal details of processing nonauditory stimuli in the human auditory cortex remain unclear, owing to the low temporal and spatial resolution of the adopted imaging techniques. Using intraoperative optical imaging of intrinsic signals recording techniques, detailed cortical activations within the auditory cortex in response to auditory and somatosensory stimulation were recorded from 3 intraoperative anesthetized patients with brain tumors located in the superior temporal gyrus. At both green-light (545 nm ± 13) and red-light (610 nm ± 10) illumination, the primary and secondary auditory cortices responded significantly to the somatosensory stimulation. As induced by the somatosensory stimulus, the average overlapping rate of the activated region was 74.51% ± 0.15, and the peak responding time occurred at poststimulus 7-8 seconds. There was no significant difference in the peak responding time between auditory and somatosensory stimuli (P<0.01, paired t test). These findings provide novel evidence for multisensory interplay within human auditory cortex at an early stage of cortical processing, which extends the understanding of multisensory mechanisms of human brain functions.

Audio-visual integration effect in lateral occipital cortex during an object recognition task: An interference pilot study.

Recent neuroimaging data support the hypothesis of a multisensory interplay at low-level sensory-specific cortex. We used an on-line interference approach by rTMS to investigate the role of the left lateral occipital cortex (LOC) in audio-visual (AV) object recognition process. Fifteen healthy volunteers performed a visual identification task of degraded pictures presented alone or simultaneously to coherent or non-coherent sounds. Focal 10-Hz rTMS at an intensity of 100% resting motor threshold was delivered simultaneously to the picture. Two blocks of 60 pictures were randomly displayed in two different experimental conditions: rTMS of the left LOC and over Cz. rTMS of the left LOC produced a worsening of the accuracy compared to rTMS over Cz specifically in the coherent AV condition. These data support the view that audio-visual interaction effect may occur at early stage of recognition processing.

Neonatal Restriction of Tactile Inputs Leads to Long-Lasting Impairments of Cross-Modal Processing.

Optimal behavior relies on the combination of inputs from multiple senses through complex interactions within neocortical networks. The ontogeny of this multisensory interplay is still unknown. Here, we identify critical factors that control the development of visual-tactile processing by combining in vivo electrophysiology with anatomical/functional assessment of cortico-cortical communication and behavioral investigation of pigmented rats. We demonstrate that the transient reduction of unimodal (tactile) inputs during a short period of neonatal development prior to the first cross-modal experience affects feed-forward subcortico-cortical interactions by attenuating the cross-modal enhancement of evoked responses in the adult primary somatosensory cortex. Moreover, the neonatal manipulation alters cortico-cortical interactions by decreasing the cross-modal synchrony and directionality in line with the sparsification of direct projections between primary somatosensory and visual cortices. At the behavioral level, these functional and structural deficits resulted in lower cross-modal matching abilities. Thus, neonatal unimodal experience during defined developmental stages is necessary for setting up the neuronal networks of multisensory processing.

141. Audiovisual integration effect in lateral occipital cortex during an object recognition task: A TMS study

Perceptual events occur in multiple sensory modalities. Recent neuroimaging data challenge the traditional models arguing for sensory integration to occur in higher-order association cortices and support the hypothesis of a multisensory interplay at low-level sensory-specific cortex. We used an on-line interference approach by rTMS to investigate the role of the left lateral occipital cortex (LOC) in audio-visual (AV) object recognition process. Fifteen healthy volunteers performed a visual identification task of degraded pictures (living and non-living) presented alone (V) or simultaneously to coherent (c-AV) or non-coherent (nc-AV) sounds. Focal 10-Hz rTMS at an intensity of 100% resting motor threshold was delivered simultaneously to the picture (500 ms). Two blocks of 60 pictures were randomly displayed in two different experimental conditions: rTMS of the left LOC and over Cz (cortical control area). Accuracy and reaction time of identification were analyzed. rTMS of the left LOC produced a worsening of the accuracy compared to rTMS over Cz specifically in the c-AV condition. Interestingly, the effect was significant for stimuli belonging to the living semantic category. These data support the view that audiovisual interaction effect may occur at early stage of recognition processing.

Sensory Neuroscience: Taste Responses in Primary Olfactory Cortex

A new electrophysiological study in rodents demonstrates that taste-odor convergence occurs in posterior piriform olfactory cortex and calls for a reformulation of classic models of the central representation of flavor.

Task-demands and audio-visual stimulus configurations modulate neural activity in the human thalamus

Recent electrophysiological studies have reported short latency modulations in cortical regions for multisensory stimuli, thereby suggesting a subcortical, possibly thalamic origin of these modulations. Concurrently, there is an ongoing debate, whether multisensory interplay reflects automatic, bottom-up driven processes or relies on top-down influences. Here, we dissociated the effects of task set and stimulus configurations on BOLD-signals in the human thalamus with event-related functional magnetic resonance imaging (fMRI). We orthogonally manipulated temporal and spatial congruency of audio-visual stimulus configurations, while subjects judged either their temporal or spatial congruency. Voxel-based fMRI results revealed increased fMRI-signals for the temporal versus spatial task in posterior and central thalamus, respectively. A more sensitive region of interest (ROI)-analysis confirmed that the posterior thalamic nuclei showed a preference for the temporal task and central thalamic nuclei for the spatial task. Moreover, the ROI-analysis also revealed enhanced fMRI-signals for spatially incongruent stimuli in the central thalamus. Together, our results demonstrate that both audio-visual stimulus configurations and task-related processing of spatial or temporal stimulus features selectively modulate thalamic processing and thus are in a position to influence cortical processing at an early stage.

Effect of auditory input on activations in infant diverse cortical regions during audiovisual processing

A fundamental question with regard to perceptual development is how multisensory information is processed in the brain during the early stages of development. Although a growing body of evidence has shown the early emergence of modality-specific functional differentiation of the cortical regions, the interplay between sensory inputs from different modalities in the developing brain is not well understood. To study the effects of auditory input during audio-visual processing in 3-month-old infants, we evaluated the spatiotemporal cortical hemodynamic responses of 50 infants while they perceived visual objects with or without accompanying sounds. The responses were measured using 94-channel near-infrared spectroscopy over the occipital, temporal, and frontal cortices. The effects of sound manipulation were pervasive throughout the diverse cortical regions and were specific to each cortical region. Visual stimuli co-occurring with sound induced the early-onset activation of the early auditory region, followed by activation of the other regions. Removal of the sound stimulus resulted in focal deactivation in the auditory regions and reduced activation in the early visual region, the association region of the temporal and parietal cortices, and the anterior prefrontal regions, suggesting multisensory interplay. In contrast, equivalent activations were observed in the lateral occipital and lateral prefrontal regions, regardless of sound manipulation. Our findings indicate that auditory input did not generally enhance overall activation in relation to visual perception, but rather induced specific changes in each cortical region. The present study implies that 3-month-old infants may perceive audio-visual multisensory inputs by using the global network of functionally differentiated cortical regions.

Thalamic influences on multisensory integration.

In everyday life our brain often receives information about events and objects in the real world via several sensory modalities, because natural objects often stimulate more than one sense. These different types of information are processed in our brain along different sensory-specific pathways, but are finally integrated into a unified percept. During the last years, studies provided compelling evidence that the neural basis of multisensory integration is not restricted to higher association areas of the cortex, but can already occur at low-level stages of sensory cortical processing and even in subcortical structures. In this article we will review the potential role of several thalamic structures in multisensory interplay and discuss their extensive anatomical connections with sensory-specific and multisensory cortical structures. We conclude that sensory-specific thalamic structures may act as a crucial processing node of multisensory interplay in addition to their traditional role as sensory relaying structure.

Do Sensory Cortices Process More than One Sensory Modality during Perceptual Judgments?

Recent studies have reported that sensory cortices process more than one sensory modality, challenging the long-lasting concept that they process only one. However, both the identity of these multimodal responses and whether they contribute to perceptual judgments is unclear. We recorded from single neurons in somatosensory cortices and primary auditory cortex while trained monkeys discriminated, on interleaved trials, either between two tactile flutter stimuli or between two acoustic flutter stimuli, and during discrimination sets that combined these two sensory modalities. We found neurons in these sensory cortices that responded to stimuli that are not of their principal sensory modality during these tasks. However, the identity of the stimulus could only be decoded from responses to their principal sensory modality during the stimulation periods and not during the processing steps that link sensation and decision making. These results suggest that multimodal encoding and perceptual judgments in these tasks occur outside the sensory cortices studied here.

Action-specific remapping of peripersonal space

Peripersonal space processing in monkeys’ brain relies on visuo-tactile neurons activated by objects near, not touching, the animal's skin. Multisensory interplay in peripersonal space is now well documented also in humans, in brain damaged patients presenting cross-modal extinction as well as in healthy subjects and typically takes the form of stronger visuo-tactile interactions in peripersonal than far space. We recently showed in healthy humans the existence of a functional link between voluntary object-oriented actions (Grasping) and the multisensory coding of the space around us (as indexed by visual–tactile interaction). Here, we investigated whether performing different actions towards the same object implies differential modulations of peripersonal space. Healthy subjects were asked to either grasp or point towards a target object. In addition, they discriminated whether tactile stimuli were delivered on their right index finger (up), or thumb (down), while ignoring visual distractors. Visuo-tactile interaction was probed in baseline Static conditions (before the movement) and in dynamic conditions (action onset and execution). Results showed that, compared to the Static baseline both actions similarly strengthened visuo-tactile interaction at the action onset, when Grasping and Pointing were kinematically indistinguishable. Crucially, Grasping induced further enhancement than Pointing in the execution phase, i.e., when the two actions kinematically diverged. These findings reveal that performing actions induce a continuous remapping of the multisensory peripersonal space as a function of on-line sensory–motor requirements, thus supporting the hypothesis of a role for peripersonal space in the motor control of voluntary actions.

Multisensory Integration: A Functional Role for Inter-Area Synchronization?

It has been suggested that synchronization of neural activity in distinct areas of the brain is important in the 'perceptual binding' of the various features relating to a specific object. A recent study has provided new evidence that inter-area synchronization is important in multisensory integration.

Changing Reference Frames during the Encoding of Tactile Events

The mindless act of swatting a mosquito on the hand poses a remarkable challenge for the brain. Given that the primary somatosensory cortex maps skin location independently of arm posture [1, 2], the brain must realign tactile coordinates in order to locate the origin of the stimuli in extrapersonal space. Previous studies have highlighted the behavioral relevance of such an external mapping of touch, which results from combining somatosensory input with proprioceptive and visual cues about body posture [3-7]. However, despite the widely held assumption about the existence of this remapping process from somatotopic to external space and various findings indirectly suggesting its consequences [8-11], a demonstration of its changing time course and nature was lacking. We examined the temporal course of this multisensory interaction and its implications for tactile awareness in humans using a crossmodal cueing paradigm [12, 13]. What we show is that before tactile events are referred to external locations [12-15], a fleeting, unconscious image of the tactile sensation abiding to a somatotopic frame of reference rules performance. We propose that this early somatotopic "glimpse" arises from the initial feed-forward sweep of neural activity to the primary somatosensory cortex, whereas the later externally-based, conscious experience reflects the activity of a somatosensory network involving recurrent connections from association areas.

Integration of Bimodal Looming Signals through Neuronal Coherence in the Temporal Lobe

The ability to integrate information across multiple sensory systems offers several behavioral advantages, from quicker reaction times and more accurate responses to better detection and more robust learning. At the neural level, multisensory integration requires large-scale interactions between different brain regions--the convergence of information from separate sensory modalities, represented by distinct neuronal populations. The interactions between these neuronal populations must be fast and flexible, so that behaviorally relevant signals belonging to the same object or event can be immediately integrated and integration of unrelated signals can be prevented. Looming signals are a particular class of signals that are behaviorally relevant for animals and that occur in both the auditory and visual domain. These signals indicate the rapid approach of objects and provide highly salient warning cues about impending impact. We show here that multisensory integration of auditory and visual looming signals may be mediated by functional interactions between auditory cortex and the superior temporal sulcus, two areas involved in integrating behaviorally relevant auditory-visual signals. Audiovisual looming signals elicited increased gamma-band coherence between these areas, relative to unimodal or receding-motion signals. This suggests that the neocortex uses fast, flexible intercortical interactions to mediate multisensory integration.