Cross-modal Integration Research Articles

Crossmodal representation of a functional robotic hand arises after extensive training in healthy participants

The way in which humans represent their own bodies is critical in guiding their interactions with the environment. To achieve successful body–space interactions, the body representation is strictly connected with that of the space immediately surrounding it through efficient visuo-tactile crossmodal integration. Such a body–space integrated representation is not fixed, but can be dynamically modulated by the use of external tools. Our study aims to explore the effect of using a complex tool, namely a functional prosthesis, on crossmodal visuo-tactile spatial interactions in healthy participants. By using the crossmodal visuo-tactile congruency paradigm, we found that prolonged training with a mechanical hand capable of distal hand movements and providing sensory feedback induces a pattern of interference, which is not observed after a brief training, between visual stimuli close to the prosthesis and touches on the body. These results suggest that after extensive, but not short, training the functional prosthesis acquires a visuo-tactile crossmodal representation akin to real limbs. This finding adds to previous evidence for the embodiment of functional prostheses in amputees, and shows that their use may also improve the crossmodal combination of somatosensory feedback delivered by the prosthesis with visual stimuli in the space around it, thus effectively augmenting the patients' visuomotor abilities.

Preserved discrimination performance and neural processing during crossmodal attention in aging.

In a recent study in younger adults (19-29 year olds) we showed evidence that distributed audiovisual attention resulted in improved discrimination performance for audiovisual stimuli compared to focused visual attention. Here, we extend our findings to healthy older adults (60-90 year olds), showing that performance benefits of distributed audiovisual attention in this population match those of younger adults. Specifically, improved performance was revealed in faster response times for semantically congruent audiovisual stimuli during distributed relative to focused visual attention, without any differences in accuracy. For semantically incongruent stimuli, discrimination accuracy was significantly improved during distributed relative to focused attention. Furthermore, event-related neural processing showed intact crossmodal integration in higher performing older adults similar to younger adults. Thus, there was insufficient evidence to support an age-related deficit in crossmodal attention.

Touch Influences Visual Perception with a Tight Orientation-Tuning

Stimuli from different sensory modalities are thought to be processed initially in distinct unisensory brain areas prior to convergence in multisensory areas. However, signals in one modality can influence the processing of signals from other modalities and recent studies suggest this cross-modal influence may occur early on, even in ‘unisensory’ areas. Some recent psychophysical studies have shown specific cross-modal effects between touch and vision during binocular rivalry, but these cannot completely rule out a response bias. To test for genuine cross-modal integration of haptic and visual signals, we investigated whether congruent haptic input could influence visual contrast sensitivity compared to incongruent haptic input in three psychophysical experiments using a two-interval, two-alternative forced-choice method to eliminate response bias. The initial experiment demonstrated that contrast thresholds for a visual grating were lower when exploring a haptic grating that shared the same orientation compared to an orthogonal orientation. Two subsequent experiments mapped the orientation and spatial frequency tunings for the congruent haptic facilitation of vision, finding a clear orientation tuning effect but not a spatial frequency tuning. In addition to an increased contrast sensitivity for iso-oriented visual-haptic gratings, we found a significant loss of sensitivity for orthogonally oriented visual-haptic gratings. We conclude that the tactile influence on vision is a result of a tactile input to orientation-tuned visual areas.

Crossmodal Integration of Conspecific Vocalizations in Rhesus Macaques

Crossmodal integration of audio/visual information is vital for recognition, interpretation and appropriate reaction to social signals. Here we examined how rhesus macaques process bimodal species-specific vocalizations by eye tracking, using an unconstrained preferential looking paradigm. Six adult rhesus monkeys (3M, 3F) were presented two side-by-side videos of unknown male conspecifics emitting different vocalizations, accompanied by the audio signal corresponding to one of the videos. The percentage of time animals looked to each video was used to assess crossmodal integration ability and the percentages of time spent looking at each of the six a priori ROIs (eyes, mouth, and rest of each video) were used to characterize scanning patterns. Animals looked more to the congruent video, confirming reports that rhesus monkeys spontaneously integrate conspecific vocalizations. Scanning patterns showed that monkeys preferentially attended to the eyes and mouth of the stimuli, with subtle differences between males and females such that females showed a tendency to differentiate the eye and mouth regions more than males. These results were similar to studies in humans indicating that when asked to assess emotion-related aspects of visual speech, people preferentially attend to the eyes. Thus, the tendency for female monkeys to show a greater differentiation between the eye and mouth regions than males may indicate that female monkeys were slightly more sensitive to the socio-emotional content of complex signals than male monkeys. The current results emphasize the importance of considering both the sex of the observer and individual variability in passive viewing behavior in nonhuman primate research.

Unconscious Presentation of Fearful Face Modulates Electrophysiological Responses to Emotional Prosody

Cross-modal integration of visual and auditory emotional cues is supposed to be advantageous in the accurate recognition of emotional signals. However, the neural locus of cross-modal integration between affective prosody and unconsciously presented facial expression in the neurologically intact population is still elusive at this point. The present study examined the influences of unconsciously presented facial expressions on the event-related potentials (ERPs) in emotional prosody recognition. In the experiment, fearful, happy, and neutral faces were presented without awareness by continuous flash suppression simultaneously with voices containing laughter and a fearful shout. The conventional peak analysis revealed that the ERPs were modulated interactively by emotional prosody and facial expression at multiple latency ranges, indicating that audio-visual integration of emotional signals takes place automatically without conscious awareness. In addition, the global field power during the late-latency range was larger for shout than for laughter only when a fearful face was presented unconsciously. The neural locus of this effect was localized to the left posterior fusiform gyrus, giving support to the view that the cortical region, traditionally considered to be unisensory region for visual processing, functions as the locus of audiovisual integration of emotional signals.

Concurrent emotional pictures modulate temporal order judgments of spatially separated audio-tactile stimuli

Although attention can be captured toward high-arousal stimuli, little is known about how perceiving emotion in one modality influences the temporal processing of non-emotional stimuli in other modalities. We addressed this issue by presenting observers spatially uninformative emotional pictures while they performed an audio-tactile temporal-order judgment (TOJ) task. In Experiment 1, audio-tactile stimuli were presented at the same location straight ahead of the participants, who had to judge “which modality came first?”. In Experiments 2 and 3, the audio-tactile stimuli were delivered one to the left and the other to the right side, and participants had to judge “which side came first?”. We found both negative and positive high-arousal pictures to significantly bias TOJs towards the tactile and away from the auditory event when the audio-tactile stimuli were spatially separated; by contrast, there was no such bias when the audio-tactile stimuli originated from the same location. To further examine whether this bias is attributable to the emotional meanings conveyed by the pictures or to their high arousal effect, we compared and contrasted the influences of near-body threat vs. remote threat (emotional) pictures on audio-tactile TOJs in Experiment 3. The bias manifested only in the near-body threat condition. Taken together, the findings indicate that visual stimuli conveying meanings of near-body interaction activate a sensorimotor functional link prioritizing the processing of tactile over auditory signals when these signals are spatially separated. In contrast, audio-tactile signals from the same location engender strong crossmodal integration, thus counteracting modality-based attentional shifts induced by the emotional pictures.

Crossmodal Integration Enhances Neural Representation of Task-Relevant Features in Audiovisual Face Perception

Previous studies have shown that audiovisual integration improves identification performance and enhances neural activity in heteromodal brain areas, for example, the posterior superior temporal sulcus/middle temporal gyrus (pSTS/MTG). Furthermore, it has also been demonstrated that attention plays an important role in crossmodal integration. In this study, we considered crossmodal integration in audiovisual facial perception and explored its effect on the neural representation of features. The audiovisual stimuli in the experiment consisted of facial movie clips that could be classified into 2 gender categories (male vs. female) or 2 emotion categories (crying vs. laughing). The visual/auditory-only stimuli were created from these movie clips by removing the auditory/visual contents. The subjects needed to make a judgment about the gender/emotion category for each movie clip in the audiovisual, visual-only, or auditory-only stimulus condition as functional magnetic resonance imaging (fMRI) signals were recorded. The neural representation of the gender/emotion feature was assessed using the decoding accuracy and the brain pattern-related reproducibility indices, obtained by a multivariate pattern analysis method from the fMRI data. In comparison to the visual-only and auditory-only stimulus conditions, we found that audiovisual integration enhanced the neural representation of task-relevant features and that feature-selective attention might play a role of modulation in the audiovisual integration.

Visuohaptic Crossmodal Matching: A Developmental fMRI Study

Research on the development of crossmodal (intersensory) perception of visual and haptic inputs suggests that communication within sensory systems develops earlier than communication across sensory systems. These behavioral changes reflect the development of neural substrates, but little is known about the neural mechanisms that underlie the developmental progression of haptic perception or visuohaptic interactions. To address this gap in knowledge, BOLD fMRI was measured during intramodal (visual-to-visual, haptic-to-haptic) and crossmodal (visual-to-haptic, haptic-to-visual) delayed match-to-sample recognition tasks with novel objects in children aged 7-8.5 years and in young adults. Tasks were further divided into encoding (sample) and matching (test) phases. During matching, adults and children demonstrated similar results. Whole-brain comparisons of intramodal and crossmodal object matching to rest revealed the network of known visuohaptic multisensory substrates, including the lateral occipital complex (LOC) and intraparietal sulcus (IPS). Comparisons of crossmodal versus intramodal conditions (i.e., (HV>VV) ͪ5; (HV>HH), (VH>VV) ͪ5; (VH>HH)) produced significant clusters in the same regions, and further, indicated specific effects of crossmodal directionality. In particular, haptic-to-visual matching tasks activated ventral regions (LOC) in both groups, while visual-to-haptic matching tasks activated dorsal regions (IPS) in adults only. Moreover, in the regions of interest as defined by contrasts with rest, crossmodal matching showed greater activation than intramodal matching. Specifically, results revealed this pattern in LOC during haptic-to-visual matching and in IPS during visual-to-haptic matching. Overall, children were highly similar to adults with the exception of some developmental differences in IPS. These results converge with existing findings on the multisensory nature of the IPS and LOC in adults, and extend our knowledge of the development of haptic perception and crossmodal integration. Critically, the results highlight the dominance of crossmodal over intramodal processing, as well as the different influences of test modality on the dorsal (IPS) and ventral (LOC) sites for visuohaptic object processing. Meeting abstract presented at VSS 2013

Movement under uncertainty: The effects of the rubber-hand illusion vary along the nonclinical autism spectrum

Recent research has begun to investigate sensory processing in relation to nonclinical variation in traits associated with the autism spectrum disorders (ASD). We propose that existing accounts of autistic perception can be augmented by considering a role for individual differences in top–down expectations for the precision of sensory input, related to the processing of state-dependent levels of uncertainty. We therefore examined ASD-like traits in relation to the rubber-hand illusion: an experimental paradigm that typically elicits crossmodal integration of visual, tactile, and proprioceptive information in an unusual illusory context. Individuals with higher ASD-like traits showed reduced effects of the rubber-hand illusion on perceived arm position and reach-to-grasp movements, compared to individuals with lower ASD-like traits. These differences occurred despite both groups reporting the typical subjective experience of the illusion concerning visuotactile integration and ownership for the rubber hand. Together these results suggest that the integration of proprioceptive information with cues for arm position derived from the illusory context differs between individuals partly in relation to traits associated with ASD. We suggest that the observed differences in sensory integration can be best explained in terms of differing expectations regarding the precision of sensory estimates in contexts that suggest uncertainty.

Cross-Modal Sensory Integration of Visual-Tactile Motion Information: Instrument Design and Human Psychophysics

Information obtained from multiple sensory modalities, such as vision and touch, is integrated to yield a holistic percept. As a haptic approach usually involves cross-modal sensory experiences, it is necessary to develop an apparatus that can characterize how a biological system integrates visual-tactile sensory information as well as how a robotic device infers object information emanating from both vision and touch. In the present study, we develop a novel visual-tactile cross-modal integration stimulator that consists of an LED panel to present visual stimuli and a tactile stimulator with three degrees of freedom that can present tactile motion stimuli with arbitrary motion direction, speed, and indentation depth in the skin. The apparatus can present cross-modal stimuli in which the spatial locations of visual and tactile stimulations are perfectly aligned. We presented visual-tactile stimuli in which the visual and tactile directions were either congruent or incongruent, and human observers reported the perceived visual direction of motion. Results showed that perceived direction of visual motion can be biased by the direction of tactile motion when visual signals are weakened. The results also showed that the visual-tactile motion integration follows the rule of temporal congruency of multi-modal inputs, a fundamental property known for cross-modal integration.

Temporally pre-presented lipreading cues release speech from informational masking

Listeners can use temporally pre-presented content cues and concurrently presented lipreading cues to improve speech recognition under masking conditions. This study investigated whether temporally pre-presented lipreading cues also unmask speech. In a test trial, before the target sentence was co-presented with the masker, either target-matched (priming) lipreading video or static face (priming-control) video was presented in quiet. Participants' target-recognition performance was improved by a shift from the priming-control condition to the priming condition when the masker was speech but not noise. This release from informational masking suggests a combined effect of working memory and cross-modal integration on selective attention to target speech.

Neural correlates of cross-modal affective priming by music in Williams syndrome

Emotional connection is the main reason people engage with music, and the emotional features of music can influence processing in other domains. Williams syndrome (WS) is a neurodevelopmental genetic disorder where musicality and sociability are prominent aspects of the phenotype. This study examined oscillatory brain activity during a musical affective priming paradigm. Participants with WS and age-matched typically developing controls heard brief emotional musical excerpts or emotionally neutral sounds and then reported the emotional valence (happy/sad) of subsequently presented faces. Participants with WS demonstrated greater evoked fronto-central alpha activity to the happy vs sad musical excerpts. The size of these alpha effects correlated with parent-reported emotional reactivity to music. Although participant groups did not differ in accuracy of identifying facial emotions, reaction time data revealed a music priming effect only in persons with WS, who responded faster when the face matched the emotional valence of the preceding musical excerpt vs when the valence differed. Matching emotional valence was also associated with greater evoked gamma activity thought to reflect cross-modal integration. This effect was not present in controls. The results suggest a specific connection between music and socioemotional processing and have implications for clinical and educational approaches for WS.

Cross-modal integration in the brain is related to phonological awareness only in typical readers, not in those with reading difficulty.

Fluent reading requires successfully mapping between visual orthographic and auditory phonological representations and is thus an intrinsically cross-modal process, though reading difficulty has often been characterized as a phonological deficit. However, recent evidence suggests that orthographic information influences phonological processing in typical developing (TD) readers, but that this effect may be blunted in those with reading difficulty (RD), suggesting that the core deficit underlying reading difficulties may be a failure to integrate orthographic and phonological information. Twenty-six (13 TD and 13 RD) children between 8 and 13 years of age participated in a functional magnetic resonance imaging (fMRI) experiment designed to assess the role of phonemic awareness in cross-modal processing. Participants completed a rhyme judgment task for word pairs presented unimodally (auditory only) and cross-modally (auditory followed by visual). For typically developing children, correlations between elision and neural activation were found for the cross-modal but not unimodal task, whereas in children with RD, no correlation was found. The results suggest that elision taps both phonemic awareness and cross-modal integration in typically developing readers, and that these processes are decoupled in children with reading difficulty.

Gone in a flash: manipulation of audiovisual temporal integration using transcranial magnetic stimulation

While converging evidence implicates the right inferior parietal lobule in audiovisual integration, its role has not been fully elucidated by direct manipulation of cortical activity. Replicating and extending an experiment initially reported by Kamke et al. (2012), we employed the sound-induced flash illusion, in which a single visual flash, when accompanied by two auditory tones, is misperceived as multiple flashes (Wilson, 1987; Shams et al., 2000). Slow repetitive (1 Hz) TMS administered to the right angular gyrus, but not the right supramarginal gyrus, induced a transient decrease in the Peak Perceived Flashes (PPF), reflecting reduced susceptibility to the illusion. This finding independently confirms that perturbation of networks involved in multisensory integration can result in a more veridical representation of asynchronous auditory and visual events and that cross-modal integration is an active process in which the objective is the identification of a meaningful constellation of inputs, at times at the expense of accuracy.

Is one enough? The case for non-additive influences of visual features on crossmodal Stroop interference

When different perceptual signals arising from the same physical entity are integrated, they form a more reliable sensory estimate. When such repetitive sensory signals are pitted against other competing stimuli, such as in a Stroop Task, this redundancy may lead to stronger processing that biases behavior toward reporting the redundant stimuli. This bias would therefore, be expected to evoke greater incongruency effects than if these stimuli did not contain redundant sensory features. In the present paper we report that this is not the case for a set of three crossmodal, auditory-visual Stroop tasks. In these tasks participants attended to, and reported, either the visual or the auditory stimulus (in separate blocks) while ignoring the other, unattended modality. The visual component of these stimuli could be purely semantic (words), purely perceptual (colors), or the combination of both. Based on previous work showing enhanced crossmodal integration and visual search gains for redundantly coded stimuli, we had expected that relative to the single features, redundant visual features would have induced both greater visual distracter incongruency effects for attended auditory targets, and been less influenced by auditory distracters for attended visual targets. Overall, reaction times were faster for visual targets and were dominated by behavioral facilitation for the cross-modal interactions (relative to interference), but showed surprisingly little influence of visual feature redundancy. Post-hoc analyses revealed modest and trending evidence for possible increases in behavioral interference for redundant visual distracters on auditory targets, however, these effects were substantially smaller than anticipated and were not accompanied by a redundancy effect for behavioral facilitation or for attended visual targets.

Cocaine Users Manifest Impaired Prosodic and Cross-Modal Emotion Processing

Background: A small number of previous studies have provided evidence that cocaine users (CU) exhibit impairments in complex social cognition tasks, while the more basic facial emotion recognition is widely unaffected. However, prosody and cross-modal emotion processing has not been systematically investigated in CU so far. Therefore, the aim of the present study was to assess complex multisensory emotion processing in CU in comparison to controls and to examine a potential association with drug use patterns.Method: The abbreviated version of the comprehensive affect testing system (CATS-A) was used to measure emotion perception across the three channels of facial affect, prosody, and semantic content in 58 CU and 48 healthy control (HC) subjects who were matched for age, sex, verbal intelligence, and years of education.Results: CU had significantly lower scores than controls in the quotient scales of “emotion recognition” and “prosody recognition” and the subtests “conflicting prosody/meaning – attend to prosody” and “match emotional prosody to emotional face” either requiring to attend to prosody or to integrate cross-modal information. In contrast, no group difference emerged for the “affect recognition quotient.” Cumulative cocaine doses and duration of cocaine use correlated negatively with emotion processing.Conclusion: CU show impaired cross-modal integration of different emotion processing channels particularly with regard to prosody, whereas more basic aspects of emotion processing such as facial affect perception are comparable to the performance of HC.

Cross-modal integration of emotions in the chemical senses

Although the brain structures involved in integrating odorant and trigeminal stimuli are well-documented, there is still a need to clarify (1) how emotional response is represented in the human brain during cross-modal interaction between odors and trigeminal stimuli, and (2) whether the degree of congruency between the two types of stimuli influences these emotional responses and their neural processing. These questions were explored combining psychophysics, event-related potentials (ERP) and fMRI in the same group of 17 subjects under a “congruent condition” (intranasal carbon dioxide mixed with the smell of orange, a combination found in soda drinks, for example), and an “incongruent condition” (intranasal carbon dioxide mixed with the smell of rose, a combination not encountered in everyday life). Responses to the 3 constituent stimuli (carbon dioxide, orange, and rose) were also measured. Hedonic and intensity ratings were collected for all stimulations. The congruent bimodal stimulus was rated as more pleasant than the incongruent. This behavioral effect was associated with enhanced neural activity in the hippocampus and anterior cingulate gyrus, indicating that these brain areas mediate reactivation of pleasant and congruent olfactory-trigeminal associations.

Impact of the spatial congruence of redundant targets on within-modal and cross-modal integration

Although the topic of sensory integration has raised increasing interest, the differing behavioral outcome of combining unisensory versus multisensory inputs has surprisingly only been scarcely investigated. In the present experiment, observers were required to respond as fast as possible to (1) lateralized visual or tactile targets presented alone, (2) double stimulation within the same modality or (3) double stimulation across modalities. Each combination was either delivered within the same hemispace (spatially aligned) or in different hemispaces (spatially misaligned). Results show that the redundancy gains (RG) obtained from the cross-modal conditions were far greater than those obtained from combinations of two visual or two tactile targets. Consistently, we observed that the reaction time distributions of cross-modal targets, but not those of within-modal targets, surpass the predicted reaction time distribution based on the summed probability distributions of each constituent stimulus presented alone. Moreover, we found that the spatial alignment of the targets did not influence the RG obtained in cross-modal conditions, whereas within-modal stimuli produced a greater RG when the targets where delivered in separate hemispaces. These results suggest that within-modal and cross-modal integration are not only distinguishable by the amount of facilitation they produce, but also by the spatial configuration under which this facilitation occurs. Our study strongly supports the notion that estimates of the same event that are more independent produce enhanced integrative gains.

Basic Multisensory Functions Can Be Acquired After Congenital Visual Pattern Deprivation in Humans

People treated for bilateral congenital cataracts offer a model to study the influence of visual deprivation in early infancy on visual and multisensory development. We investigated cross-modal integration capabilities in cataract patients using a simple detection task that provided redundant information to two different senses. In both patients and controls, redundancy gains were consistent with coactivation models, indicating an integrated processing of modality-specific information. This finding is in contrast with recent studies showing impaired higher-level multisensory interactions in cataract patients. The present results suggest that basic cross-modal integrative processes for simple short stimuli do not depend on visual and/or crossmodal input since birth.

Fingerprints of developmental dyslexia

Abstract Developmental dyslexia is a reading disorder—but what are the underlying cognitive and neurobiological mechanisms? The variety of different explanations in the literature ranges from phonological abilities via sensory (auditory, visual) processing to attention, learning, and cross-modal integration. Here, we suggest that it is advantageous to accept the variability encountered in dyslexia. Different cognitive profiles, or “fingerprints”, characterise groups of dyslexic individuals. Most importantly, these fingerprints have a repercussion in differential brain activation patterns in dyslexia, supporting the notion of a neurobiological basis. We conclude that considering the neurocognitive fingerprint of a dyslexic reader may be helpful to plan an individually targeted training, which, in turn, may optimise the outcome of remediation.