Crossmodal Correspondences Research Articles

Crossmodal correspondence of elevation/pitch and size/pitch is driven by real-world features.

Crossmodal correspondences are consistent associations between sensory features from different modalities, with some theories suggesting they may either reflect environmental correlations or stem from innate neural structures. This study investigates this question by examining whether retinotopic or representational features of stimuli induce crossmodal congruency effects. Participants completed an auditory pitch discrimination task paired with visual stimuli varying in their sensory (retinotopic) or representational (scene integrated) nature, for both the elevation/pitch and size/pitch correspondences. Results show that only representational visual stimuli produced crossmodal congruency effects on pitch discrimination. These results support an environmental statistics hypothesis, suggesting crossmodal correspondences rely on real-world features rather than on sensory representations.

Брендинг в постмодернистском контексте: звуковые логотипы как культурные маркеры

The aim of the study is to identify the peculiarities of sound logos perception and their role in shaping brand cultural identity in the context of postmodern culture. The article analyzes the interaction between audio and visual brand elements and their impact on consumers’ emotional perception. The significance of cross-modal correspondences and sound symbolism in creating effective audio assets is examined. The scientific novelty lies in developing a multicultural cognitive theory of sound symbolism perception, integrating universal cognitive mechanisms and cultural specificities. As a result, the study reveals the complex role of sound in creating brand cultural identity in the era of globalization and proposes new approaches to developing sound strategies for international brands.

It’s Relative! The Cross-Modal Effects of Music Density on Perception of Product Size

Individuals inevitably make inferences concerning size when they consider an object presented without cues to its actual size. Recent studies show that structural differences in background music can influence consumers’ perceptions of product attributes through cross-modal correspondence. We introduce a new structural element of music, the “music event rate.” To this end, we make a distinction between the absolute density (notes per time unit) and the relative density (notes per measure or music event rate) and propose an influence of the latter on consumers’ perceptions of product size. Our study employs a 2 (notes per measure: four vs. eight) × 2 (music tempo: slow vs. fast) × 2 (music mode: major vs. minor) between-subject experimental design. Dependent variables are participants’ size estimates of three different food items (pictures of burger, pizza, wine bottle) presented online. The results show that the number of notes per measure has an influence on consumers’ size perception of food items insofar as a higher event rate or number of notes per measure in a music stimulus leads consumers to infer a smaller product size, whereas a lower number of notes per measure leads consumers to perceive products as larger. This cross-modal effects of the music event rate on product size perception represent a previously unexplored influence on consumer behavior.

Perceiving depth beyond sight: Evaluating intrinsic and learned cues via a proof of concept sensory substitution method in the visually impaired and sighted.

This study explores spatial perception of depth by employing a novel proof of concept sensory substitution algorithm. The algorithm taps into existing cognitive scaffolds such as language and cross modal correspondences by naming objects in the scene while representing their elevation and depth by manipulation of the auditory properties for each axis. While the representation of verticality utilized a previously tested correspondence with pitch, the representation of depth employed an ecologically inspired manipulation, based on the loss of gain and filtration of higher frequency sounds over distance. The study, involving 40 participants, seven of which were blind (5) or visually impaired (2), investigates the intrinsicness of an ecologically inspired mapping of auditory cues for depth by comparing it to an interchanged condition where the mappings of the two axes are swapped. All participants successfully learned to use the algorithm following a very brief period of training, with the blind and visually impaired participants showing similar levels of success for learning to use the algorithm as did their sighted counterparts. A significant difference was found at baseline between the two conditions, indicating the intuitiveness of the original ecologically inspired mapping. Despite this, participants were able to achieve similar success rates following the training in both conditions. The findings indicate that both intrinsic and learned cues come into play with respect to depth perception. Moreover, they suggest that by employing perceptual learning, novel sensory mappings can be trained in adulthood. Regarding the blind and visually impaired, the results also support the convergence view, which claims that with training, their spatial abilities can converge with those of the sighted. Finally, we discuss how the algorithm can open new avenues for accessibility technologies, virtual reality, and other practical applications.

Visual and auditory stimulus features, and their crossmodal correspondence, affects perceptual selection in the bounce/stream illusion

Visual and auditory stimulus features, and their crossmodal correspondence, affects perceptual selection in the bounce/stream illusion

The role of audiovisual congruence in aesthetic appreciation of contemporary music and visual art

Does congruence between auditory and visual modalities affect aesthetic experience? While cross-modal correspondences between vision and hearing are well-documented, previous studies show conflicting results regarding whether audiovisual correspondence affects subjective aesthetic experience. Here, in collaboration with the Kentler International Drawing Space (NYC, USA), we depart from previous research by using music specifically composed to pair with visual art in the professionally-curated Music as Image and Metaphor exhibition. Our pre-registered online experiment consisted of 4 conditions: Audio, Visual, Audio-Visual-Intended (artist-intended pairing of art/music), and Audio-Visual-Random (random shuffling). Participants (N = 201) were presented with 16 pieces and could click to proceed to the next piece whenever they liked. We used time spent as an implicit index of aesthetic interest. Additionally, after each piece, participants were asked about their subjective experience (e.g., feeling moved). We found that participants spent significantly more time with Audio, followed by Audiovisual, followed by Visual pieces; however, they felt most moved in the Audiovisual (bi-modal) conditions. Ratings of audiovisual correspondence were significantly higher for the Audiovisual-Intended compared to Audiovisual-Random condition; interestingly, though, there were no significant differences between intended and random conditions on any other subjective rating scale, or for time spent. Collectively, these results call into question the relationship between cross-modal correspondence and aesthetic appreciation. Additionally, the results complicate the use of time spent as an implicit measure of aesthetic experience.

Beyond Sensory Boundaries: Innovative Applications of Synesthetic Design in Shaping Deep Brand Perceptions and Experiences

With the development of modern technology and concepts, the object of design expands from material form to non-material form, and the product modeling design gradually develops towards all-round perception and experience. Products are not only the carrier of function, but also the medium of emotional communication. In an era of information overload and short consumer attention spans, building deep and lasting brand perceptions has become a major challenge for marketers. The researcher explores the innovative application of synesthetic design principles, clarifies the role of synesthetic imagery interoperability and design expressions, constructs a design translation path of synesthetic imagery interoperability with the product as the medium and emotion as the dominant, and explores the process and verification method of the application of synesthetic imagery interoperability in the product styling design, so that synesthetic design can be used as a strategic tool that transcends the boundaries of the traditional sensory marketing, and thus cultivates deep brand experience and cognition. We use the generic design as a strategic tool to transcend the boundaries of traditional sensory marketing, thus cultivating profound brand experience and cognition. By integrating multi-sensory experiences and utilizing the inherent cross-modal correspondences between different sensory modalities, synesthetic design facilitates comprehensive and immersive brand interactions. Through a combination of theoretical exploration and empirical analysis, the researchers demonstrate how synesthetic design can significantly enhance brand memorability, differentiation and emotional resonance. It further explores how the experience of empathy influences consumer behavior and brand loyalty through underlying psychological mechanisms. The results emphasize the potential of synesthetic design in building deeper and more intuitive connections between consumers and brands. The application of synesthetic imagery interoperability to product styling design can elevate users' sensory experience to emotional experience through the use of synesthesia, which provides a new path for product styling design application research.

The effect of rosé wine colors on expected flavor and tastiness: A cross-modal correspondence explanation

How and why do wines with different colors influence consumer responses? Despite strong evidence from research on cross-modal correspondences that food and drink colors influence flavor perceptions and subsequent consumer responses, it is unclear from the wine literature whether consumers prefer paler or slightly darker colors of rosé. The aim of this research is twofold: (1) identify from an existing set of (relatively pale) prototypical color shades of Rosé de Provence what specific color (paler vs. slightly darker) leads to more positive consumer responses and (2) shed light on the underlying mechanisms that explain this relationship. Using a large sample of French wine consumers (n = 601), we test our hypotheses via a within-subjects design experiment. We first examined the effect of five shades of Rosé de Provence wines (nacre, light melon, sand, melon, and peach) on consumer responses (willingness to pay, willingness to buy, color liking). To further investigate the underlying roles of fruity flavor and expected tastiness (in that order), we conducted a series of two-condition within-subjects serial mediations. Our findings consistently show more positive consumer responses for Rosés de Provence with peach and melon shades (i.e., slightly darker colors) compared to nacre, light melon, and sand shades (i.e., paler colors). Further, a stronger association between slightly darker colors and fruity flavor enhances expected tastiness, which leads to more positive consumer responses. Our findings provide key theoretical contributions and managerial implications by suggesting consumer responses for Rosés de Provence and corresponding explanations based on cross-modal correspondences.

Are you colour deaf?

Originating from antiquity, the idea of associating colour with music has been researched extensively in recent decades. The terms for this phenomenon include crossmodal correspondences and synaesthesia (or chromaesthesia), both of which refer to associations our brain makes from stimuli that it perceives through different senses. Correspondence between sound and music, and light and colour, has been a scholarly topic for years—mostly from a scientific point of view. This thesis aims to investigate different views on the subject, focusing on its artistic/aesthetic rather than neurobiological components. Music-colour correspondence was born from the need of philosophers to make sense of both music and the world. Linguistics has proven ambiguous when used to explain or make sense out of music, hence colour has been a very powerful replacement. It is possible to draw parallels between sound and light because of their similar ontological nature (vibration). The goal of this thesis is to prove that such an association can enhance a classical music performance for the audience (as related to engagement) and for the performer (as related to analysis, artistic input). As a case study, Olivier Messiaen’s Theme and Variations is analysed in this rather unconventional colour-coded way. Keywords: synaesthesia, chromaesthesia, crossmodal correspondences, Olivier Messiaen, colour and music.

Colour/shape-taste correspondences across three languages in ChatGPT

Crossmodal correspondences, the tendency for a sensory feature / attribute in one sensory modality (either physically present or merely imagined), to be associated with a sensory feature in another sensory modality, have been studied extensively, revealing consistent patterns, such as sweet tastes being associated with pink colours and round shapes across languages. The present research explores whether such correspondences are captured by ChatGPT, a large language model developed by OpenAI. Across twelve studies, this research investigates colour/shapes-taste crossmodal correspondences in ChatGPT-3.5 and -4o, focusing on associations between shapes/colours and the five basic tastes across three languages (English, Japanese, and Spanish). Studies 1A-F examined taste-shape associations, using prompts in three languages to assess ChatGPT's association of round and angular shapes with the five basic tastes. The results indicated significant, consistent, associations between shape and taste, with, for example, round shapes strongly associated with sweet/umami tastes and angular shapes with bitter/salty/sour tastes. The magnitude of shape-taste matching appears to be greater in ChatGPT-4o than in ChatGPT-3.5, and ChatGPT prompted in English and Spanish than ChatGPT prompted in Japanese. Studies 2A-F focused on colour-taste correspondences, using ChatGPT to assess associations between eleven colours and the five basic tastes. The results indicated that ChatGPT-4o, but not ChatGPT-3.5, generally replicates the patterns of colour-taste correspondences that have previously been observed in human participants. Specifically, ChatGPT-4o associates sweet tastes with pink, sour with yellow, salty with white/blue, bitter with black, and umami with red across languages. However, the magnitude/similarity of shape/colour-taste matching observed in ChatGPT-4o appears to be more pronounced (i.e., having little variance, large mean difference), which does not adequately reflect the subtle nuances typically seen in human shape/colour-taste correspondences. These findings suggest that ChatGPT captures colour/shapes-taste correspondences, with language- and GPT version-specific variations, albeit with some differences when compared to previous studies involving human participants. These findings contribute valuable knowledge to the field of crossmodal correspondences, explore the possibility of generative AI that resembles human perceptual systems and cognition across languages, and provide insight into the development and evolution of generative AI systems that capture human crossmodal correspondences.

Cross-modal correspondence study between odors and shapes of traditional Chinese incense

Cross-modal correspondence study between odors and shapes of traditional Chinese incense

Crossmodal Correspondence Mediates Crossmodal Transfer from Visual to Auditory Stimuli in Category Learning.

This article investigated whether crossmodal correspondence, as a sensory translation phenomenon, can mediate crossmodal transfer from visual to auditory stimuli in category learning and whether multimodal category learning can influence the crossmodal correspondence between auditory and visual stimuli. Experiment 1 showed that the category knowledge acquired from elevation stimuli affected the categorization of pitch stimuli when there were robust crossmodal correspondence effects between elevation and size, indicating that crossmodal transfer occurred between elevation and pitch stimuli. Experiments 2 and 3 revealed that the size category knowledge could not be transferred to the categorization of pitches, but interestingly, size and pitch category learning determined the direction of the pitch-size correspondence, suggesting that the pitch-size correspondence was not stable and could be determined using multimodal category learning. Experiment 4 provided further evidence that there was no crossmodal transfer between size and pitch, due to the absence of a robust pitch-size correspondence. These results demonstrated that crossmodal transfer can occur between audio-visual stimuli with crossmodal correspondence, and multisensory category learning can change the corresponding relationship between audio-visual stimuli. These findings suggest that crossmodal transfer and crossmodal correspondence share similar abstract representations, which can be mediated by semantic content such as category labels.

Progressive discrepancy elimination for visible–infrared person re-identification

Visible–Infrared Person Re-Identification (VI-ReID) aims to retrieve the identities of persons from different modalities. The significant distribution discrepancy makes this task challenging. Recent methods usually generate an intermediate modality to bridge the modality gap. However, these methods may lead to spectral confusion, and ignore the negative impact of visible channel redundancy information on establishing cross-modal correspondence. In this paper, we propose a Progressive Discrepancy Elimination Model (PDEM) for VI-ReID. Specifically, we design a Single-Channel Stripe Aggregation (SCSA) module to synthesize high-quality transitional modality images by using only raw visible and infrared spectral stripes. Meanwhile, we propose a two-stage Spectral Information Filtering (SIF) strategy, consisting of our designed Semantic Consistency Loss (SCL) and Cascade Aggregation Loss (CAL). SCL and CAL are employed in different training stages to achieve visible task-related information filtering and visible–infrared multi-scale alignment, respectively. In this way, the most relevant semantics with infrared modality in visible features are retained, and spectral correspondence is subsequently learned in a small cross-modal gap. We conduct extensive experiments on two VI-ReID datasets and achieve superior performance compared to most state-of-the-art methods. The source code of this paper is available at https://github.com/wxz0530/PDEM.

Scent-driven Selective Attention on Gambling Outcome: Implications for Responsible Gambling.

Many casinos diffuse a pleasant ambient scent into their facilities as a customer experience management practice, but the ethics of this scenting process is questionable. Although the effect of a pleasant scent on cognitive, emotional, and behavioral responses has been well-documented, its effect on attention during gambling has yet to be explored. Grounded in the tenets of the top-down control of attention and cross-modal correspondence between vision and olfaction, we conduct two eye-tracking experiments that involve different electronic casino games including video slots and live Cussec. The findings consistently show that pleasant ambient scent prolongs attention and induces more frequent attention to the win/loss areas on the video screen. The findings add to the implications related to responsible gambling by inspiring the stakeholders to consider the use of ambient scent in the gambling environment. Theoretically, the findings offer insights into scent as the catalyst that directs attention to goal-related information, while scent and goal do not need to be congruent in traits.

Old and new versions of the Molyneux question: A review of experimental answers

The ‘Molyneux problem’ is typically framed in terms of the crossmodal matching of shape information from touch to vision. Indeed, shape along with intensity have commonly been considered amodal stimulus properties/dimensions (at least by developmental researchers). However, it is important to note that what is common, if anything, to the senses differs in the two cases: It is the physical stimulus (and possibly also the associated phenomenology) that is thought to be the same in the case of crossmodal (or intermodal) shape matching between touch and vision, whereas it is the nature of the underlying neural encoding that is said to be similar in the case of crossmodal matching of auditory and visual stimulus intensity. While the first empirical data to have been published on these two forms of putatively amodal crossmodal matching appeared to suggest that they both emerge surprisingly early in the course of human development (i.e., within the first month of life), certain of these seminal findings have proved difficult to replicate. Ultimately, therefore, there is currently little convincing evidence to support the notion that such putatively innate crossmodal matching of amodal stimulus dimensions is actually different in kind from the various other crossmodal correspondences that are seemingly acquired at various points during the course of human development (typically as a result of the internalization of the crossmodal statistics of the environment). As such, there may be nothing particularly special about the type of crossmodal matching thought to underlie the ‘Molyneux problem’, and continued interest in the issue may inadvertently have helped to sustain a misguided account of the differences between different types of crossmodal correspondence.

Crossmodal correspondence between lexical tones and visual motions: a forced-choice mapping task on Mandarin Chinese

Abstract Crossmodal correspondence refers to the phenomenon in which individuals match stimulus features (e.g., auditory pitch) with different sensory modalities (e.g., visual size). While studies on correspondences exhibited by suprasegmentals have mostly focused on pitch-size and pitch-shape associations, audiospatial binding observed in the production and perception of Mandarin tones, where pitch of the syllable distinguishes word meanings, sheds light on the symbolic potential of auditory pitch. In the present study, a forced-choice mapping task was conducted in the form of a word guessing game, where native Mandarin listeners select the meaning of an auditory “alien” word from two visual motions. The results showed that: (1) listeners reliably match auditory tones with visual motions in the way that pitch trajectories are congruent with spatial movements, (2) vowel category impacts tone-motion correspondence when syllables are articulated in non-contour tones, and (3) the capacities in driving the tone-motion correspondence are different across tonal categories. These findings further contribute to our understanding of the sound symbolic potential of lexical tones and expand the boundary of crossmodal correspondence that can be demonstrated by pitch.

Design Complexity: Assessing Cross-Modal Correspondence between Complex Food Images and the Desire to Eat.

Design Complexity: Assessing Cross-Modal Correspondence between Complex Food Images and the Desire to Eat.

Speeded classification of visual events is sensitive to crossmodal intensity correspondence.

Crossmodal correspondences refer to systematic associations between stimulus attributes encountered in different sensory modalities. These correspondences can be probed in the speeded classification task where they tend to produce congruency effects. This study aimed to replicate and extend previous work conducted by Marks (1987, Experiment 3, Journal of Experimental Psychology: Human Perception and Performance, Vol. 13, No. 3, 384-394) which demonstrated a crossmodal correspondence between auditory and visual intensity attributes. Experiment 1 successfully replicates Marks' original finding that performance in a brightness classification task is affected by whether the loudness of a concurrently presented auditory distractor matches the brightness of the visual target. Furthermore, in line with the original study, we found that this effect was absent in a lightness classification task. In Experiment 2, we demonstrate that loudness-brightness correspondence is robust even when the exact stimulus input changes. This finding suggests that there is a context-dependent mapping between loudness and brightness levels, rather than an absolute mapping between any particular intensity levels. Finally, exploratory analysis using the diffusion model for conflict tasks indicated that evidence from the task-irrelevant modality generates a burst of weak, short-lived automatic activation that can bias decision-making in difficult tasks, but not in easy tasks. Our results provide further evidence for the existence of a flexible crossmodal correspondence between brightness and loudness, which might be helpful in determining one's distance to a stimulus source during the early stages of multisensory integration. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Dissociable neuronal mechanism for different crossmodal correspondence effects in humans.

Crossmodal correspondences (CMCs) refer to associations between seemingly arbitrary stimulus features in different sensory modalities. Pitch‑size correspondences refer to the strong association of e.g., small objects with high pitches. Pitch‑elevation correspondences refer to the strong association of e.g., visuospatial elevated objects with high pitches. We used functional magnetic resonance imaging (fMRI) to study the neural components, which underlie the CMCs in pitch‑size and spatial pitch‑elevation. This study focuses on answering the question of whether or not different CMCs are driven by similar neural mechanisms. The comparison of congruent against incongruent trials allows the estimation of CMC effects across different CMCs. The analysis of the measured neural activity in different CMCs strongly pointed toward different mechanisms which are involved in the processing of pitch‑size and pitch‑elevation correspondences. Differential, whole brain effects were observed within the superior parietal lobule (SPL), cerebellum and Heschls' gyrus (HG). Further, the angular gyrus (AnG), the intraparietal sulcus (IPS) and anterior cingulate cortex (ACC) were engaged in processing the CMCs but showed different effects for processing congruent compared to incongruent stimulus presentations. Within pitch‑size significant effects in the AnG and ACC were found for congruent stimulus presentations whereas for pitch‑elevation, significant effects in the ACC and IPS were found for incongruent stimulus presentations. In summary, the present results indicated differential neural processing in different simple audio‑visual CMCs.

The organization of semantic associations between senses in language

Abstract Distributional semantic representations were used to investigate crossmodal correspondences within language, offering a comprehensive analysis of how sensory experiences interconnect in linguistic constructs. By computing semantic proximity between words from different sensory modalities, a crossmodal semantic network was constructed, providing a general view of crossmodal correspondences in the English language. Community detection techniques were applied to unveil domains of experience where crossmodal correspondences were likely to manifest, while also considering the role of affective dimensions in shaping these domains. The study revealed the existence of an architecture of structured domains of experience in language, whereby crossmodal correspondences are deeply embedded. The present research highlights the roles of emotion and statistical associations in the organization of sensory concepts across modalities in language. The domains identified, including food, the body, the physical world and emotions/values, underscored the intricate interplay between the senses, emotion and semantic patterns. These findings align with the embodied lexicon hypothesis and the semantic coding hypothesis, emphasizing the capacity of language to capture and reflect crossmodal correspondences’ emotional and perceptual subtleties in the form of networks, while also revealing opportunities for further perceptual research on crossmodal correspondences and multisensory integration.