Crossmodal Congruency Effects Research Articles

Involvement of the superior colliculi in crossmodal correspondences.

There is an increasing body of evidence suggesting that there are low-level perceptual processes involved in crossmodal correspondences. In this study, we investigate the involvement of the superior colliculi in three basic crossmodal correspondences: elevation/pitch, lightness/pitch, and size/pitch. Using a psychophysical design, we modulate visual input to the superior colliculus to test whether the superior colliculus is required for behavioural crossmodal congruency effects to manifest in an unspeeded multisensory discrimination task. In the elevation/pitch task, superior colliculus involvement is required for a behavioural elevation/pitch congruency effect to manifest in the task. In the lightness/pitch and size/pitch task, we observed a behavioural elevation/pitch congruency effect regardless of superior colliculus involvement. These results suggest that the elevation/pitch correspondence may be processed differently to other low-level crossmodal correspondences. The implications of a distributed model of crossmodal correspondence processing in the brain are discussed.

Addressing the Association Between Action Video Game Playing Experience and Visual Search in Naturalistic Multisensory Scenes.

Prior studies investigating the effects of routine action video game play have demonstrated improvements in a variety of cognitive processes, including improvements in attentional tasks. However, there is little evidence indicating that the cognitive benefits of playing action video games generalize from simplified unisensory stimuli to multisensory scenes- a fundamental characteristic of natural, everyday life environments. The present study addressed if video game experience has an impact on crossmodal congruency effects when searching through such multisensory scenes. We compared the performance of action video game players (AVGPs) and non-video game players (NVGPs) on a visual search task for objects embedded in video clips of realistic scenes. We conducted two identical online experiments with gender-balanced samples, for a total of N = 130. Overall, the data replicated previous findings reporting search benefits when visual targets were accompanied by semantically congruent auditory events, compared to neutral or incongruent ones. However, according to the results, AVGPs did not consistently outperform NVGPs in the overall search task, nor did they use multisensory cues more efficiently than NVGPs. Exploratory analyses with self-reported gender as a variable revealed a potential difference in response strategy between experienced male and female AVGPs when dealing with crossmodal cues. These findings suggest that the generalization of the advantage of AVG experience to realistic, crossmodal situations should be made with caution and considering gender-related issues.

Neural basis of sound-symbolic pseudoword-shape correspondences

Non-arbitrary mapping between the sound of a word and its meaning, termed sound symbolism, is commonly studied through crossmodal correspondences between sounds and visual shapes, e.g., auditory pseudowords, like ‘mohloh’ and ‘kehteh’, are matched to rounded and pointed visual shapes, respectively. Here, we used functional magnetic resonance imaging (fMRI) during a crossmodal matching task to investigate the hypotheses that sound symbolism (1) involves language processing; (2) depends on multisensory integration; (3) reflects embodiment of speech in hand movements. These hypotheses lead to corresponding neuroanatomical predictions of crossmodal congruency effects in (1) the language network; (2) areas mediating multisensory processing, including visual and auditory cortex; (3) regions responsible for sensorimotor control of the hand and mouth. Right-handed participants (n = 22) encountered audiovisual stimuli comprising a simultaneously presented visual shape (rounded or pointed) and an auditory pseudoword (‘mohloh’ or ‘kehteh’) and indicated via a right-hand keypress whether the stimuli matched or not. Reaction times were faster for congruent than incongruent stimuli. Univariate analysis showed that activity was greater for the congruent compared to the incongruent condition in the left primary and association auditory cortex, and left anterior fusiform/parahippocampal gyri. Multivoxel pattern analysis revealed higher classification accuracy for the audiovisual stimuli when congruent than when incongruent, in the pars opercularis of the left inferior frontal (Broca's area), the left supramarginal, and the right mid-occipital gyri. These findings, considered in relation to the neuroanatomical predictions, support the first two hypotheses and suggest that sound symbolism involves both language processing and multisensory integration.

Empathy as a predictor of peripersonal space: Evidence from the crossmodal congruency task

To investigate whether individual differences in Empathy predict the characteristics of Peripersonal Space (PPS) representations, we asked participants to complete the IRI questionnaire and a visuo-tactile crossmodal congruency task (CCT) as an index of PPS. In the CCT, they responded to the elevation of a tactile target while ignoring a visual distractor presented at the same (i.e. congruent) or different (i.e. incongruent) elevation. The target-distractor distance was also manipulated in depth, with visual distractors randomly presented at near, middle or far locations (0 cm, 25 cm or 50 cm). The near and middle crossmodal congruency effects (CCE) were inversely related to participants’ scores on the Empathic Concern sub-scale (EC). Furthermore, the slope of participants’ CCE across locations was related to EC scores, with flatter slopes for higher EC individuals. Thus, higher EC individuals showed reduced visuo-tactile integration responses within PPS and a reduced differentiation between PPS and extra-personal space (EPS).

Crossmodal congruency effects between sound and food pictures in a forced-choice task.

Research using short musical sequences and musical tracks created by means of computer algorithms has demonstrated that individuals with or without musical skills can match these soundtracks to specific tastes with above-chance accuracy. More recently, a study that investigated implicit effects associated with crossmodal congruency/incongruency between auditory cues and food images found that such soundtracks are effective in eliciting facilitating effects of taste quality classification with congruent food images as well. In the present study, we tested whether this crossmodal congruency between auditory cues and food images may also influence food image choice by means of a forced-choice task. We selected and used sweet and salty soundtracks as stimuli and food images including both low- and high-calorie exemplars and asked participants to select which food they would prefer to eat (one sweet and one salty) while listening to the soundtracks. We found a general greater proportion of food choices in the soundtracks matching tastes conditions, and that soundtracks matching tastes are effective in influencing congruent food image choices, supporting previous research and adding new interesting outcomes.

Crossmodal Congruency Enhances Performance of Healthy Older Adults in Visual-Tactile Pattern Matching.

One of the pivotal challenges of aging is to maintain independence in the activities of daily life. In order to adapt to changes in the environment, it is crucial to continuously process and accurately combine simultaneous input from different sensory systems, i.e., crossmodal or multisensory integration. With aging, performance decreases in multiple domains, affecting bottom-up sensory processing as well as top-down control. However, whether this decline leads to impairments in crossmodal interactions remains an unresolved question. While some researchers propose that crossmodal interactions degrade with age, others suggest that they are conserved or even gain compensatory importance. To address this question, we compared the behavioral performance of older and young participants in a well-established crossmodal matching task, requiring the evaluation of congruency in simultaneously presented visual and tactile patterns. Older participants performed significantly worse than young controls in the crossmodal task when being stimulated at their individual unimodal visual and tactile perception thresholds. Performance increased with adjustment of stimulus intensities. This improvement was driven by better detection of congruent stimulus pairs, while the detection of incongruent pairs was not significantly enhanced. These results indicate that age-related impairments lead to poor performance in complex crossmodal scenarios and demanding cognitive tasks. Crossmodal congruency effects attenuate the difficulties of older adults in visuotactile pattern matching and might be an important factor to drive the benefits of older adults demonstrated in various crossmodal integration scenarios. Congruency effects might, therefore, be used to develop strategies for cognitive training and neurological rehabilitation.

Overt spatial attention modulates multisensory selection.

In daily life, signals from the different senses are often integrated to enhance multisensory perception. However, an important, yet currently still controversial, topic concerns the need for attention in this integration process. To investigate this question, we turned to the processing of multisensory distractors. Note that multisensory target processing is typically confounded with attention as people attend to the stimuli that they respond to. We therefore designed a multisensory flanker task in which the target and distractor stimuli were both multisensory and the congruency between the features (auditory and visual) was varied orthogonally. In addition, we manipulated whether distractor or target was within the focus of participants' gaze (i.e., was overtly attended). Importantly, distractor congruency effects were modulated by this manipulation. Fixating the distractor led to crossmodal congruency effects between the visual and auditory feature dimensions (e.g., a visually incongruent distractor interfered more if it was also auditorily incongruent with the target), while congruency effects were independent of each other when the distractor was not fixated (i.e., visual interference was not modulated by auditory interference in this case). These results suggest that distractors outside the focus of overt attention are processed at the level of features whereas those distractors presented at fixation are processed as a configuration of features. Taken together, these results can be taken to suggest that the multisensory integration of irrelevant stimuli depends on the focus of spatial attention. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Audiovisual semantic interactions between linguistic and nonlinguistic stimuli: The time-courses and categorical specificity.

We examined the time-courses and categorical specificity of the crossmodal semantic congruency effects elicited by naturalistic sounds and spoken words on the processing of visual pictures (Experiment 1) and printed words (Experiment 2). Auditory cues were presented at 7 different stimulus onset asynchronies (SOAs) with respect to the visual targets, and participants made speeded categorization judgments (living vs. nonliving). Three common effects were observed across 2 experiments: Both naturalistic sounds and spoken words induced a slowly emerging congruency effect when leading by 250 ms or more in the congruent compared with the incongruent condition, and a rapidly emerging inhibitory effect when leading by 250 ms or less in the incongruent condition as opposed to the noise condition. Only spoken words that did not match the visual targets elicited an additional inhibitory effect when leading by 100 ms or when presented simultaneously. Compared with nonlinguistic stimuli, the crossmodal congruency effects associated with linguistic stimuli occurred over a wider range of SOAs and occurred at a more specific level of the category hierarchy (i.e., the basic level) than was required by the task. A comprehensive framework is proposed to provide a dynamic view regarding how meaning is extracted during the processing of visual or auditory linguistic and nonlinguistic stimuli, therefore contributing to our understanding of multisensory semantic processing in humans. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

Shifts in target modality cause attentional reset: Evidence from sequential modulation of crossmodal congruency effects.

Reduced congruency effects after a preceding incongruent trial suggest a conflict-monitoring process, which reactively triggers the recruitment of attentional control in subsequent trials. In the present study, we assessed this sequential modulation of crossmodal congruency effects separately in two different tasks. Participants performed a location judgment task and a numerical judgment task in a block-wise fashion in a modality-switching paradigm. Stimuli were presented simultaneously in two modalities and were either congruent or incongruent (e.g., left visual object, right sound) with each other. The target modality was indicated by a cue, so that the target modalities either repeated or switched in successive trials. For both tasks, the results indicated reduced congruency effects after an incongruent trial only for modality repetitions, but not for switches. This finding suggests that modality switches induce a shift in episodic context, which in turn leads to an attentional reset. This reset eliminates the sequential modulation of congruency effects.

Force feedback facilitates multisensory integration during robotic tool use

The present study investigated the effects of force feedback in relation to tool use on the multisensory integration of visuo-tactile information. Participants learned to control a robotic tool through a surgical robotic interface. Following tool-use training, participants performed a crossmodal congruency task, by responding to tactile vibrations applied to their hands, while ignoring visual distractors superimposed on the robotic tools. In the first experiment it was found that tool-use training with force feedback facilitates multisensory integration of signals from the tool, as reflected in a stronger crossmodal congruency effect with the force feedback training compared to training without force feedback and to no training. The second experiment extends these findings by showing that training with realistic online force feedback resulted in a stronger crossmodal congruency effect compared to training in which force feedback was delayed. The present study highlights the importance of haptic information for multisensory integration and extends findings from classical tool-use studies to the domain of robotic tools. We argue that such crossmodal congruency effects are an objective measure of robotic tool integration and propose some potential applications in surgical robotics, robotic tools, and human-tool interaction.

Visuo‐tactile integration and body ownership during self‐generated action

Although there is increasing knowledge about how visual and tactile cues from the hands are integrated, little is known about how self-generated hand movements affect such multisensory integration. Visuo-tactile integration often occurs under highly dynamic conditions requiring sensorimotor updating. Here, we quantified visuo-tactile integration by measuring cross-modal congruency effects (CCEs) in different bimanual hand movement conditions with the use of a robotic platform. We found that classical CCEs also occurred during bimanual self-generated hand movements, and that such movements lowered the magnitude of visuo-tactile CCEs as compared to static conditions. Visuo-tactile integration, body ownership and the sense of agency were decreased by adding a temporal visuo-motor delay between hand movements and visual feedback. These data show that visual stimuli interfere less with the perception of tactile stimuli during movement than during static conditions, especially when decoupled from predictive motor information. The results suggest that current models of visuo-tactile integration need to be extended to account for multisensory integration in dynamic conditions.

Multisensory integration across the menstrual cycle

Evidence suggests that spatial processing changes across time in naturally cycling women, which is likely due to neuromodulatory effects of steroid hormones. Yet, it is unknown whether crossmodal spatial processes depend on steroid hormones as well. In the present experiment, the crossmodal congruency task was used to assess visuo-tactile interactions in naturally cycling women, women using hormonal contraceptives and men. Participants adopted either a crossed or uncrossed hands posture. It was tested whether a postural effect of hand crossing on multisensory interactions in the crossmodal congruency task is modulated by women's cycle phase. We found that visuotactile interactions changed according to cycle phase. Naturally cycling women showed a significant difference between the menstrual and the luteal phase for crossed, but not for uncrossed hands postures. The two control groups showed no test sessions effects. Regression analysis revealed a positive relation between estradiol levels and the size of crossmodal congruency effects (CCE), indicating that estradiol seems to have a neuromodulatory effect on posture processing.

Extending the Body to Virtual Tools Using a Robotic Surgical Interface: Evidence from the Crossmodal Congruency Task

The effects of real-world tool use on body or space representations are relatively well established in cognitive neuroscience. Several studies have shown, for example, that active tool use results in a facilitated integration of multisensory information in peripersonal space, i.e. the space directly surrounding the body. However, it remains unknown to what extent similar mechanisms apply to the use of virtual-robotic tools, such as those used in the field of surgical robotics, in which a surgeon may use bimanual haptic interfaces to control a surgery robot at a remote location. This paper presents two experiments in which participants used a haptic handle, originally designed for a commercial surgery robot, to control a virtual tool. The integration of multisensory information related to the virtual-robotic tool was assessed by means of the crossmodal congruency task, in which subjects responded to tactile vibrations applied to their fingers while ignoring visual distractors superimposed on the tip of the virtual-robotic tool. Our results show that active virtual-robotic tool use changes the spatial modulation of the crossmodal congruency effects, comparable to changes in the representation of peripersonal space observed during real-world tool use. Moreover, when the virtual-robotic tools were held in a crossed position, the visual distractors interfered strongly with tactile stimuli that was connected with the hand via the tool, reflecting a remapping of peripersonal space. Such remapping was not only observed when the virtual-robotic tools were actively used (Experiment 1), but also when passively held the tools (Experiment 2). The present study extends earlier findings on the extension of peripersonal space from physical and pointing tools to virtual-robotic tools using techniques from haptics and virtual reality. We discuss our data with respect to learning and human factors in the field of surgical robotics and discuss the use of new technologies in the field of cognitive neuroscience.

Crossmodal correspondences

For more than a century now, researchers have acknowledged the existence of crossmodal congruency effects between dimensions of sensory stimuli in the general (i.e., non-synesthetic) population. Such phenomena, known by a variety of terms including ‘crossmodal correspondences’, involve individual stimulus properties, rely on a crossmodal mapping of unisensory features, and appear to be shared by the majority of individuals. Over the last few years, a number of studies have shed light on many key aspects of crossmodal correspondences, ranging from their role in multisensory integration, their developmental trajectories, their occurrence in non-human mammals, their neural underpinning and the role of learning. I will present a brief overview of the latest findings on crossmodal correspondences, highlight standing questions and provide direction for future research.

Interplay of Multisensory Processing, Attention, and Consciousness as Revealed by Bistable Figures

We examined the novel crossmodal semantic congruency effect on bistable figures in which a static stimulus gives rise to two competing percepts that alternate over time. Participants viewed the bistable figure “my wife or my mother-in-law” while listening to the voice of an old woman or a young lady speaking in an unfamiliar language. They had to report whether they saw the old woman, the young lady, or a mixed percept. Robust crossmodal semantic congruency effects in the measures of the first percept and the predominance duration were observed. The possibilities that the participants simply responded to, and/or that they fixed at the location in favor of, the percept congruent with the sound that they happened to hear were ruled out. When the participants were instructed to maintain their attention to a specific view, a strong top-down modulation on the perception of bistable figure was observed, although the audiovisual semantic congruency effect still remained. These results thus demonstrate that top-d...

Leg muscle vibration modulates bodily self-consciousness: integration of proprioceptive, visual, and tactile signals

Behavioral studies have used visuo-tactile conflicts between a participant's body and a visually presented fake or virtual body to investigate the importance of bodily perception for self-consciousness (bodily self-consciousness). Illusory self-identification with a fake body and changes in tactile processing--modulation of visuo-tactile cross-modal congruency effects (CCEs)--were reported in previous findings. Although proprioceptive signals are deemed important for bodily self-consciousness, their contribution to the representation of the full body has not been studied. Here we investigated whether and how self-identification and tactile processing (CCE magnitude) could be modified by altering proprioceptive signals with 80-Hz vibrations at the legs. Participants made elevation judgments of tactile cues (while ignoring nearby lights) during synchronous and asynchronous stroking of a seen fake body. We found that proprioceptive signals during vibrations altered the magnitude of self-identification and mislocalization of touch (CCE) in a synchrony-dependent fashion: we observed an increase of self-identification and CCE magnitude during asynchronous stroking. In a second control experiment we studied whether proprioceptive signals per se, or those from the lower limbs in particular, were essential for these changes. We applied vibrations at the upper limbs (which provide no information about the position of the participant's body in space) and in this case observed no modulation of bodily self-consciousness or tactile perception. These data link proprioceptive signals from the legs that are conveyed through the dorsal column-medial lemniscal pathway to bodily self-consciousness. We discuss their integration with bodily signals from vision and touch for full-body representations.

Crossmodal congruency effects based on stimulus identity

We investigated whether the crossmodal congruency effect, normally observed in spatial compatibility tasks, would also affect performance on a task based on discriminating stimulus identity (i.e., a non-spatial dimension). Two rhythms were presented to participants' eyes, ears, and/or hands in a 4-alternative rhythm discrimination task. Stimulus identity and stimulus modality were varied orthogonally. When the target and distractor rhythms were presented in different sensory modalities, significant crossmodal congruency effects were observed in all conditions (i.e., performance on the incongruent distractor trials was significantly more error-prone than on the congruent distractor trials). In contrast to the results of previous studies, these crossmodal distractor effects were neither based on the spatial compatibility of the stimuli nor on an abstract semantic matching of stimulus identity, but instead on the identity of the target rhythm. Intriguingly, the magnitude of the crossmodal congruency effects differed as a function of the target modality, but were unaffected by the modality of the distractor.

Seeing the body modulates audiotactile integration

Audiotactile integration has been studied using various experimental setups but so far crossmodal congruency effects (CCEs) have not been found for tactile targets paired with auditory distractors. In the present study we investigated whether audiotactile CCEs exist and, if so, whether these CCEs have similar characteristics to those found by previous authors with visual distractors. We measured audiotactile CCEs by attaching four vibrators to the backs of participants and presented auditory stimuli from four loudspeakers placed, in separate blocks, at different distances in front of or behind the participant's body. Participants discriminated the elevation of tactile stimuli while ignoring the auditory distractors. CCEs were found only when participants were provided with noninformative vision of their own body, as seen from behind via a camera and head-mounted display; they were absent when participants did not view their body. Furthermore, in contrast to visuotactile CCEs, audiotactile CCEs did not depend on whether the distractors were presented on the same or different side as the tactile targets. The present study provides the first demonstration of an audiotactile CCE: incongruent auditory distractors impaired performance on a tactile elevation discrimination task relative to performance with congruent distractors. We show that audiotactile CCEs differ from visuotactile CCEs as they do not appear to be as sensitive to the spatial relations between the distractors and the tactile stimuli. We also show that these CCEs are modulated by vision of the body.

Keeping in touch with one's self: multisensory mechanisms of self-consciousness.

BackgroundThe spatial unity between self and body can be disrupted by employing conflicting visual-somatosensory bodily input, thereby bringing neurological observations on bodily self-consciousness under scientific scrutiny. Here we designed a novel paradigm linking the study of bodily self-consciousness to the spatial representation of visuo-tactile stimuli by measuring crossmodal congruency effects (CCEs) for the full body.Methodology/Principal FindingsWe measured full body CCEs by attaching four vibrator-light pairs to the trunks (backs) of subjects who viewed their bodies from behind via a camera and a head mounted display (HMD). Subjects made speeded elevation (up/down) judgments of the tactile stimuli while ignoring light stimuli. To modulate self-identification for the seen body subjects were stroked on their backs with a stick and the felt stroking was either synchronous or asynchronous with the stroking that could be seen via the HMD.We found that (1) tactile stimuli were mislocalized towards the seen body (2) CCEs were modulated systematically during visual-somatosensory conflict when subjects viewed their body but not when they viewed a body-sized object, i.e. CCEs were larger during synchronous than during asynchronous stroking of the body and (3) these changes in the mapping of tactile stimuli were induced in the same experimental condition in which predictable changes in bodily self-consciousness occurred.Conclusions/SignificanceThese data reveal that systematic alterations in the mapping of tactile stimuli occur in a full body illusion and thus establish CCE magnitude as an online performance proxy for subjective changes in global bodily self-consciousness.

Interactions of different body parts in peripersonal space: how vision of the foot influences tactile perception at the hand

The body schema, a constantly updated representation of the body and its parts, has been suggested to emerge from body part-specific representations which integrate tactile, visual, and proprioceptive information about the identity and posture of the body. Studies using different approaches have provided evidence for a distinct representation of the visual space approximately 30 cm around the upper body, and predominantly the hands, termed the peripersonal space. In humans, peripersonal space representations have often been investigated with a visual-tactile crossmodal congruency task. We used this task to test if a representation of peripersonal space exists also around the feet, and to explore possible interactions of peripersonal space representations of different body parts. In Experiment 1, tactile stimuli to the hands and feet were judged according to their elevation while visual distractors presented near the same limbs had to be ignored. Crossmodal congruency effects did not differ between the two types of limbs, suggesting a representation of peripersonal space also around the feet. In Experiment 2, tactile stimuli were presented to the hands, and visual distractors were flashed either near the participant's foot, near a fake foot, or in distant space. Crossmodal congruency effects were larger in the real foot condition than in the two other conditions, indicating interactions between the peripersonal space representations of foot and hand. Furthermore, results of all three conditions showed that vision of the stimulated body part, compared to only proprioceptive input about its location, strongly influences crossmodal interactions for tactile perception, affirming the central role of vision in the construction of the body schema.