Crossmodal Congruency Research Articles

Crossmodal correspondence of elevation/pitch and size/pitch is driven by real-world features.

Crossmodal correspondences are consistent associations between sensory features from different modalities, with some theories suggesting they may either reflect environmental correlations or stem from innate neural structures. This study investigates this question by examining whether retinotopic or representational features of stimuli induce crossmodal congruency effects. Participants completed an auditory pitch discrimination task paired with visual stimuli varying in their sensory (retinotopic) or representational (scene integrated) nature, for both the elevation/pitch and size/pitch correspondences. Results show that only representational visual stimuli produced crossmodal congruency effects on pitch discrimination. These results support an environmental statistics hypothesis, suggesting crossmodal correspondences rely on real-world features rather than on sensory representations.

Impact of aging on crossmodal attention switching

Previous studies on crossmodal visual-auditory attention switching using a spatial discrimination task showed performance costs when the target modality changed relative to when it repeated. The present study (n = 42 for each age group) examined age-related changes in crossmodal attention switching by asking young (age range 19 to 30 years old) and older (age range 64 to 80 years old) participants to respond to unimodal central cues and bimodal lateralized stimuli. The participants’ task was to indicate the location of the target in the relevant modality using button presses. Results showed general attention switch costs. Additionally, we found no specific age-related increase of attention switch costs (no difference in performance between switch and repetition of target modality), but age-related increased mixing costs (decreased performance for repetition in modality-mixed condition compared to single target modality). Moreover, spatial distraction produced a crossmodal congruency effect, which was only slightly larger in older adults. Taken together, age-related increased mixing costs suggest a general difficulty with maintaining more than one task, but no specific age-related crossmodal impairment in crossmodal attention switching.

Involvement of the superior colliculi in crossmodal correspondences.

There is an increasing body of evidence suggesting that there are low-level perceptual processes involved in crossmodal correspondences. In this study, we investigate the involvement of the superior colliculi in three basic crossmodal correspondences: elevation/pitch, lightness/pitch, and size/pitch. Using a psychophysical design, we modulate visual input to the superior colliculus to test whether the superior colliculus is required for behavioural crossmodal congruency effects to manifest in an unspeeded multisensory discrimination task. In the elevation/pitch task, superior colliculus involvement is required for a behavioural elevation/pitch congruency effect to manifest in the task. In the lightness/pitch and size/pitch task, we observed a behavioural elevation/pitch congruency effect regardless of superior colliculus involvement. These results suggest that the elevation/pitch correspondence may be processed differently to other low-level crossmodal correspondences. The implications of a distributed model of crossmodal correspondence processing in the brain are discussed.

Addressing the Association Between Action Video Game Playing Experience and Visual Search in Naturalistic Multisensory Scenes.

Prior studies investigating the effects of routine action video game play have demonstrated improvements in a variety of cognitive processes, including improvements in attentional tasks. However, there is little evidence indicating that the cognitive benefits of playing action video games generalize from simplified unisensory stimuli to multisensory scenes- a fundamental characteristic of natural, everyday life environments. The present study addressed if video game experience has an impact on crossmodal congruency effects when searching through such multisensory scenes. We compared the performance of action video game players (AVGPs) and non-video game players (NVGPs) on a visual search task for objects embedded in video clips of realistic scenes. We conducted two identical online experiments with gender-balanced samples, for a total of N = 130. Overall, the data replicated previous findings reporting search benefits when visual targets were accompanied by semantically congruent auditory events, compared to neutral or incongruent ones. However, according to the results, AVGPs did not consistently outperform NVGPs in the overall search task, nor did they use multisensory cues more efficiently than NVGPs. Exploratory analyses with self-reported gender as a variable revealed a potential difference in response strategy between experienced male and female AVGPs when dealing with crossmodal cues. These findings suggest that the generalization of the advantage of AVG experience to realistic, crossmodal situations should be made with caution and considering gender-related issues.

Sound-to-Touch Crossmodal Pitch Matching for Short Sounds.

This article explores how to relate sound and touch in terms of their spectral characteristics based on crossmodal congruence. The context is the audio-to-tactile conversion of short sounds frequently used for user experience improvement across various applications. For each short sound, a single-frequency amplitude-modulated vibration is synthesized so that their intensive and temporal characteristics are very similar. It leaves the vibration frequency, which determines the tactile pitch, as the only variable. Each sound is paired with many vibrations of different frequencies. The congruence between sound and vibration is evaluated for 175 pairs (25 sounds × 7 vibration frequencies). This dataset is employed to estimate a functional relationship from the sound loudness spectrum of sound to the most harmonious vibration frequency. Finally, this sound-to-touch crossmodal pitch mapping function is evaluated using cross-validation. To our knowledge, this is the first attempt to find general rules for spectral matching between sound and touch.

The left–right reversed visual feedback of the hand affects multisensory interaction within peripersonal space

The interaction between vision and touch, known as the crossmodal congruency effect, has been extensively investigated in several research studies. Recent studies have revealed that the crossmodal congruency effect involves body representations. However, it is unclear how bodily information (e.g., location, posture, motion) is linked to visual and tactile inputs. Three experiments were conducted to investigate this issue. In Experiment 1, participants performed a crossmodal congruency task in which both their hand appearance and the motor trajectories were left–right reversed. The results showed that the crossmodal congruency effect was not observed in the reversal condition, whereas participants showed significant crossmodal congruency in the control condition, in which there was no visual manipulation of the hand. In Experiments 2 and 3, where either the hand appearance or motor trajectory was left–right reversed individually, a significant crossmodal congruency effect was observed. This study demonstrated that visual manipulation of hand appearance and motor trajectories both affected the crossmodal congruency effect, although neither showed a dominant effect that solely altered the crossmodal congruency effect. The present results provide insights into the relationship between visual-tactile interactions and bodily information.

Harmony of senses: Exploring the impact of sound aesthetic features' on taste imagery

People reliably associate visual aesthetic features such as curvature and symmetry with tastes. In the present study, considering the transitive hypothesis of crossmodal correspondences, we evaluated whether these findings would extend to the relationship between sound aesthetic features and tastes, and whether feature-based congruency or affective priming would explain the influence of melodies on taste imagery. In Experiment 1, we evaluated how people associate different melody profiles (balanced vs. unbalanced, smooth vs. jagged, symmetrical vs. asymmetrical, simpler vs. more complex) with different tastes (sweet, sour, bitter, salty), as well as the melodies' associated fluency, valence, and arousal. Smooth and complex melodies were perceived as sweeter, jagged and unbalanced melodies as sourer, asymmetrical and jagged melodies as more bitter, and jagged and balanced melodies as saltier. In Experiment 2, we selected the most strongly associated aesthetic sound dimension with tastes, namely contour, and evaluated whether crossmodal congruency, based on crossmodal correspondences, or affective priming would influence people’s sensory and hedonic imagery associated with sweet and sour foods. In the imagery tasks, the participants showed higher sour ratings in the sourness task and higher sweetness ratings in the sweetness task. In the sweet imagery task, smooth melodies led to sweeter and less sour food imagery, whereas jagged melodies in the sourness task led to more sour and less sweet food imagery. These results favour the crossmodal congruency explanation rather than the affective alternative, which we ponder in the general discussion.

Beyond peripersonal boundaries: insights from crossmodal interactions.

We experience our self as a body located in space. However, how information about self-location is integrated into multisensory processes underlying the representation of the peripersonal space (PPS), is still unclear. Prior studies showed that the presence of visual information related to oneself modulates the multisensory processes underlying PPS. Here, we used the crossmodal congruency effect (CCE) to test whether this top-down modulation depends on the spatial location of the body-related visual information. Participants responded to tactile events on their bodies while trying to ignore a visual distractor presented on the mirror reflection of their body (Self) either in the peripersonal space (Near) or in the extrapersonal space (Far). We found larger CCE when visual events were presented on the mirror reflection in the peripersonal space, as compared to the extrapersonal space. These results suggest that top-down modulation of the multisensory bodily self is only possible within the PPS.

Effect of a Body Part Action on Body Perception of the Other Inactive Body Part

The rubber hand illusion (RHI) is an illusory experience in which a fake rubber hand is felt as if it were one’s own hand when the visible fake and invisible real hands are stimulated synchronously. Although several studies have suggested contributions of action to body ownership using the RHI paradigm, the relationship between body ownership and agency has not yet been fully revealed. To better understand this relationship, the present study investigated the transfer of body ownership between body parts induced by a body part action. Using an RHI paradigm involving virtual reality, we tested whether a simple finger action of the dominant (active) hand can induce the embodiment of a virtual hand corresponding to the non-dominant (inactive) hand. We evaluated the illusory experience, perceptual changes, and physiological changes during the experiment with a subjective questionnaire, crossmodal congruency effect measurement, and skin temperature measurement, respectively. The results demonstrated that the finger action induced the embodiment of both virtual hands, causing a significant increase in agency of the inactive hand. This suggests that the illusory experience induced by an active body part contributes to an increase in agency as well as body ownership over the virtual body of the other inactive body part.

Neural basis of sound-symbolic pseudoword-shape correspondences

Non-arbitrary mapping between the sound of a word and its meaning, termed sound symbolism, is commonly studied through crossmodal correspondences between sounds and visual shapes, e.g., auditory pseudowords, like ‘mohloh’ and ‘kehteh’, are matched to rounded and pointed visual shapes, respectively. Here, we used functional magnetic resonance imaging (fMRI) during a crossmodal matching task to investigate the hypotheses that sound symbolism (1) involves language processing; (2) depends on multisensory integration; (3) reflects embodiment of speech in hand movements. These hypotheses lead to corresponding neuroanatomical predictions of crossmodal congruency effects in (1) the language network; (2) areas mediating multisensory processing, including visual and auditory cortex; (3) regions responsible for sensorimotor control of the hand and mouth. Right-handed participants (n = 22) encountered audiovisual stimuli comprising a simultaneously presented visual shape (rounded or pointed) and an auditory pseudoword (‘mohloh’ or ‘kehteh’) and indicated via a right-hand keypress whether the stimuli matched or not. Reaction times were faster for congruent than incongruent stimuli. Univariate analysis showed that activity was greater for the congruent compared to the incongruent condition in the left primary and association auditory cortex, and left anterior fusiform/parahippocampal gyri. Multivoxel pattern analysis revealed higher classification accuracy for the audiovisual stimuli when congruent than when incongruent, in the pars opercularis of the left inferior frontal (Broca's area), the left supramarginal, and the right mid-occipital gyri. These findings, considered in relation to the neuroanatomical predictions, support the first two hypotheses and suggest that sound symbolism involves both language processing and multisensory integration.

Towards Visual-Tactile Integration of Shoulder and Hand Using Immersive Virtual Reality.

Shoulder-controlled hand neuroprostheses are wearable devices designed to assist hand function in people with cervical spinal cord injury (SCI). They use preserved shoulder movements to control artificial actuators. Due to the concurrent afferent (i.e., shoulder proprioception) and visual (i.e., hand response) feedback, these wearables may affect the user's body somatosensory representation. To investigate this effect, we propose an experimental paradigm that uses immersive virtual reality (VR) environment to emulate the use of a shoulder-controlled hand neuroprostheses and an adapted version of a visual-tactile integration task (i.e., Crossmodal Congruency Task) as an assessment tool. Data from seven non-disabled participants validates the experimental setup, with preliminary statistical analysis revealing no significant difference across the means of VR and visual-tactile integration tasks. The results serve as a proof-of-concept for the proposed paradigm, paving the way for further research with improvements in the experimental design and a larger sample size.

Impacts of sensory multimodality congruence and familiarity with short use on cosmetic product evaluation.

Cross-modal association between sensory modalities is a natural phenomenon in the perception of our environment. For cosmetic evaluation, touch and smell are the two major sensory modalities involved in the whole product perception. In this study, we investigate whether a specific cosmetic texture is preferentially associated with a specific fragrance: congruence between texture and fragrance. In addition, we investigate whether 1-week use of a fragrance-texture congruent or non-congruent product can influence user's whole product appreciation and well-being. We have conducted a four-test experiment with 29 participants; first in the laboratory to evaluate: six fragrances and four textures individually with free description (test 1); the same stimuli with a description with cross-modal descriptors (test 2); 10 fragrance-texture combined products (test 3); and secondly at home, two fragrance-texture combined products: one congruent and one non-congruent (test 4). Results showed that: (1) For a given texture type, specific olfactory notes are necessary to lead to a congruent cross-modal pairing product. (2) Sensory modal congruent products produce the highest hedonic response. (3)Real-life use or familiarisation with a product can influence not only the degree of cross-modal congruence but also overall cosmetic product appreciation.

Implicit Association Test (IAT) Studies Investigating Pitch-Shape Audiovisual Cross-modal Associations Across Language Groups.

Previous studies have shown that Chinese speakers and non-Chinese speakers exhibit different patterns of cross-modal congruence for the lexical tones of Mandarin Chinese, depending on which features of the pitch they attend to. But is this pattern of language-specific listening a conscious cultural strategy or an automatic processing effect? If automatic, does it also apply when the same pitch contours no longer sound like speech? Implicit Association Tests (IATs) provide an indirect measure of cross-modal association. In a series of IAT studies, conducted with participants with three kinds of language backgrounds (Chinese-dominant bilinguals, Chinese balanced bilinguals, and English speakers with no Chinese experience) we find language-specific congruence effects for Mandarin lexical tones but not for matched sine-wave stimuli. That is, for linguistic stimuli, non-Chinese speakers show advantages for pitch-height congruence (high-pointy, low-curvy); no congruence effects were found for Chinese speakers. For non-linguistic stimuli, all participant groups showed advantages for pitch-height congruence. The present findings suggest that non-lexical tone congruence (high-pointy, low-curvy) is a basic congruence pattern, and the acquisition of a language with lexical tone can alter this perception.

Olfactory-gustatory simultaneity judgments: A preliminary study on the congruency-dependent temporal window of multisensory binding.

A greater congruency of audio and video expands the temporal binding window (TBW). A similar phenomenon may occur with a combination of odor and taste, which are the main components of flavor. TBW is defined as the temporal resolution of synchrony perception. The larger the TBW, the lower the resolution. We hypothesized that the more congruent the odor and taste, the lower the temporal resolution of synchrony perception. To examine this hypothesis, 10 female participants performed simultaneity judgment (SJ) tasks under congruent (soy sauce odor with saline) and incongruent (cherry tree leaf odor with saline) conditions and evaluated the congruency with saltiness for the two odors. In the SJ tasks, participants reported whether odor and taste were presented simultaneously or successively. We assumed a Gaussian distribution for the temporal distributions of the simultaneous response rates and calculated the approximations. In addition, we computed the half width at half height (HWHH) as an index of TBW based on the coefficient of approximation for the temporal distribution of the simultaneous response rates. HWHH was significantly larger under congruent condition than under incongruent condition. In addition, congruency with saltiness had a significant moderate positive correlation with HWHH. The larger the HWHH, the lower the temporal resolution of synchrony perception, supporting the hypothesis. This study suggests that the width of TBW depends on the cross-modal congruency similar to the case for audio-visual SJs. However, methodological improvements, including a larger sample size and gender-independent recruitment of participants, are essential to enhance the reliability of the findings because some of the results did not provide sufficient ESs or power.

Searching for perceptual similarity within, and between, the (chemical) senses.

In this narrative historical review, I want to take a closer look at the concept of perceptual similarity both as it applies within, and between, the chemical senses (specifically taste and smell). The discussion is linked to issues of affective similarity and connotative meaning. The relation between intramodal and crossmodal judgments of perceptual similarity, and the putatively special status of those odorants that happen to take on taste qualities will also be discussed. An important distinction is drawn between the interrelated, though sometimes distinct, notions of perceptual similarity and crossmodal congruency, specifically as they relate to the comparison of chemosensory stimuli. Such phenomena are often referred to as crossmodal correspondences, or by others (incorrectly in my view), as a kind of ubiquitous synesthesia.

Embodiment of supernumerary robotic limbs in virtual reality

The supernumerary robotic limb system expands the motor function of human users by adding extra artificially designed limbs. It is important for us to embody the system as if it is a part of one’s own body and to maintain cognitive transparency in which the cognitive load is suppressed. Embodiment studies have been conducted with an expansion of bodily functions through a “substitution” and “extension”. However, there have been few studies on the “addition” of supernumerary body parts. In this study, we developed a supernumerary robotic limb system that operates in a virtual environment, and then evaluated whether the extra limb can be regarded as a part of one’s own body using a questionnaire and whether the perception of peripersonal space changes with a visuotactile crossmodal congruency task. We found that the participants can embody the extra-limbs after using the supernumerary robotic limb system. We also found a positive correlation between the perceptual change in the crossmodal congruency task and the subjective feeling that the number of one’s arms had increased (supernumerary limb sensation). These results suggest that the addition of an extra body part may cause the participants to feel that they had acquired a new body part that differs from their original body part through a functional expansion.

Peripersonal space around the upper and the lower limbs

Peripersonal space (PPS), the space closely surrounding the body, is typically characterised by enhanced multisensory integration. Neurophysiological and behavioural studies have consistently shown stronger visuo-tactile integration when a visual stimulus is presented close to the tactually stimulate body part in near space (within PPS) than in far space. However, in the majority of these studies, tactile stimuli were delivered to the upper limbs, torso and face. Therefore, it is not known whether the space surrounding the lower limbs is characterised by similar multisensory properties. To address this question, we asked participants to complete two versions of the classic visuo-tactile crossmodal congruency task in which they had to perform speeded elevation judgements of tactile stimuli presented to the dorsum of the hand and foot while a simultaneous visual distractor was presented at spatially congruent or incongruent locations either in near or far space. In line with existing evidence, when the tactile target was presented to the hand, the size of the crossmodal congruency effect (CCE) decreased in far as compared to near space, suggesting stronger visuo-tactile multisensory integration within PPS. In contrast, when the tactile target was presented to the foot, the CCE decreased for visual distractors in near than far space. These findings show systematic differences between the representation of PPS around upper and lower limbs, suggesting that the multisensory properties of the different body part-centred representations of PPS are likely to depend on the potential actions performed by the different body parts.

Crossmodal Congruency Between Background Music and the Online Store Environment: The Moderating Role of Shopping Goals.

Despite the robust evidence that congruent background music in the physical store environment positively affects consumer reactions, less is known about its effects in an online context. The present study aims (1) to examine whether congruency via multiple elicited crossmodal correspondences between background music and the online store environment (e.g., perceived lightness, loudness, and coldness of the cue/environment) leads to more positive affective, evaluative, and behavioral consumer reactions and (2) to investigate the moderating role of shopping goals on this crossmodal congruency effect. Previous research showed that low task-relevant atmospheric cues like music can have a negative effect on consumers when they visit a website with a purchase goal in mind. An online experiment was conducted with 239 respondents randomly assigned to a shopping goal (experiential browsing vs. goal-directed searching) and a music condition (no music, crossmodally congruent music, or crossmodally incongruent music). Our results show that crossmodally incongruent background music (vs. no music) leads to more positive consumer reactions for experiential browsers and more negative consumer reactions for goal-directed searchers. Conversely, crossmodally congruent background music (vs. no music) has a positive effect on experiential browsers and no adverse effect on goal-directed searchers. Additionally, the presence of crossmodally congruent background music leads to more positive consumer reactions than the presence of crossmodally incongruent background music, independent of the shopping goal. We extend previous research on multisensory congruency effects by showing the added value of establishing congruency between music and the store environment via multiple elicited crossmodal correspondences in the online environment, countering previously found negative effects of low-task relevant atmospheric cues for goal-directed searchers.

Empathy as a predictor of peripersonal space: Evidence from the crossmodal congruency task

To investigate whether individual differences in Empathy predict the characteristics of Peripersonal Space (PPS) representations, we asked participants to complete the IRI questionnaire and a visuo-tactile crossmodal congruency task (CCT) as an index of PPS. In the CCT, they responded to the elevation of a tactile target while ignoring a visual distractor presented at the same (i.e. congruent) or different (i.e. incongruent) elevation. The target-distractor distance was also manipulated in depth, with visual distractors randomly presented at near, middle or far locations (0 cm, 25 cm or 50 cm). The near and middle crossmodal congruency effects (CCE) were inversely related to participants’ scores on the Empathic Concern sub-scale (EC). Furthermore, the slope of participants’ CCE across locations was related to EC scores, with flatter slopes for higher EC individuals. Thus, higher EC individuals showed reduced visuo-tactile integration responses within PPS and a reduced differentiation between PPS and extra-personal space (EPS).

Influence of the crossmodal congruency between color and flavor on product evaluations: Evidence from behavioral and oscillatory brain responses

People often associate features or stimuli across senses, and these crossmodal correspondences also function as criteria for judging whether a pair of features or stimuli is crossmodally congruent or incongruent. We conducted two studies to examine whether the crossmodal congruency between a drink color and a flavor could also elicit more positive ratings of this drink. In Study 1, the experimental task was to simultaneously view a photo of a colored drink and to taste a beverage whose flavor was either congruent or incongruent with the color. Consequently, the color-flavor congruent drinks received more positive ratings than the color-flavor incongruent ones. In Study 2, we conducted an EEG experiment to investigate both behavioral and oscillatory brain responses to the crossmodal congruency between the successively presented color and flavor. The results of subjective ratings did not reveal any significant difference between the color-flavor congruent and incongruent stimuli. However, the time–frequency analyses on the EEG data revealed that rating the color-flavor incongruent beverages elicited greater alpha band power in the parietal region, compared to the color-flavor congruent stimuli. Collectively, these findings revealed the influence of color-flavor congruency on the subjective ratings of beverages, and demonstrated how this effect was modulated by the temporal synchrony of the color and flavor cues.