Crossmodal Correspondences Research Articles

The organization of semantic associations between senses in language

Abstract Distributional semantic representations were used to investigate crossmodal correspondences within language, offering a comprehensive analysis of how sensory experiences interconnect in linguistic constructs. By computing semantic proximity between words from different sensory modalities, a crossmodal semantic network was constructed, providing a general view of crossmodal correspondences in the English language. Community detection techniques were applied to unveil domains of experience where crossmodal correspondences were likely to manifest, while also considering the role of affective dimensions in shaping these domains. The study revealed the existence of an architecture of structured domains of experience in language, whereby crossmodal correspondences are deeply embedded. The present research highlights the roles of emotion and statistical associations in the organization of sensory concepts across modalities in language. The domains identified, including food, the body, the physical world and emotions/values, underscored the intricate interplay between the senses, emotion and semantic patterns. These findings align with the embodied lexicon hypothesis and the semantic coding hypothesis, emphasizing the capacity of language to capture and reflect crossmodal correspondences’ emotional and perceptual subtleties in the form of networks, while also revealing opportunities for further perceptual research on crossmodal correspondences and multisensory integration.

Humans (Homo sapiens) but not baboons (Papio papio) demonstrate crossmodal pitch-luminance correspondence.

Humans spontaneously and consistently map information coming from different sensory modalities. Surprisingly, the phylogenetic origin of such cross-modal correspondences has been under-investigated. A notable exception is the study of Ludwig et al. (Visuoauditory mappings between high luminance and high pitch are shared by chimpanzees [Pan troglodytes] and humans. Proceedings of the National Academy of Sciences, 108(51), 20661-20665) which reports that both humans and chimpanzees spontaneously map high-pitched sounds with bright objects and low-pitched sounds with dark objects. Our pre-registered study aimed to directly replicate this research on both humans and baboons (Papio papio), an old world monkey which is more phylogenetically distant from humans than chimpanzees. Following Ludwig et al.participants were presented with a visual classification task where they had to sort black and white square (low and high luminance), while background sounds (low or high-pitched tones) were playing. Whereas we replicated the finding that humans' performance on the visual task was affected by congruency between sound and luminance of the target, we did not find any of those effects on baboons' performance. These results question the presence of a shared cross-modal pitch-luminance mapping in other nonhuman primates.

Inducing Novel Sound-Taste Correspondences via an Associative Learning Task.

The interest in crossmodal correspondences, including those involving sounds and involving tastes, has experienced rapid growth in recent years. However, the mechanisms underlying these correspondences are not well understood. In the present study (N = 302), we used an associative learning paradigm, based on previous literature using simple sounds with no consensual taste associations (i.e., square and triangle wave sounds at 200 Hz) and taste words (i.e., sweet and bitter), to test the influence of two potential mechanisms in establishing sound-taste correspondences and investigate whether either learning mechanism could give rise to new and long-lasting associations. Specifically, we examined an emotional mediation account (i.e., using sad and happy emoji facial expressions) and a transitive path (i.e., sound-taste correspondence being mediated by color, using red and black colored squares). The results revealed that the associative learning paradigm mapping the triangle wave tone with a happy emoji facial expression induced a novel crossmodal correspondence between this sound and the word sweet. Importantly, we found that this novel association was still present two months after the experimental learning paradigm. None of the other mappings, emotional or transitive, gave rise to any significant associations between sound and taste. These findings provide evidence that new crossmodal correspondences between sounds and tastes can be created by leveraging the affective connection between both dimensions, helping elucidate the mechanisms underlying these associations. Moreover, these findings reveal that these associations can last for several weeks after the experimental session through which they were induced.

Involvement of the superior colliculi in crossmodal correspondences.

There is an increasing body of evidence suggesting that there are low-level perceptual processes involved in crossmodal correspondences. In this study, we investigate the involvement of the superior colliculi in three basic crossmodal correspondences: elevation/pitch, lightness/pitch, and size/pitch. Using a psychophysical design, we modulate visual input to the superior colliculus to test whether the superior colliculus is required for behavioural crossmodal congruency effects to manifest in an unspeeded multisensory discrimination task. In the elevation/pitch task, superior colliculus involvement is required for a behavioural elevation/pitch congruency effect to manifest in the task. In the lightness/pitch and size/pitch task, we observed a behavioural elevation/pitch congruency effect regardless of superior colliculus involvement. These results suggest that the elevation/pitch correspondence may be processed differently to other low-level crossmodal correspondences. The implications of a distributed model of crossmodal correspondence processing in the brain are discussed.

Hot and round: How temperature and shape impact the multisensory appreciation of cornmeal

Rounded shapes are significantly more liked and are typically also evaluated as warmer and sweeter than angular shapes/foods. Here, we assessed the temperature-shape crossmodal pairing for cornmeal. In a between-participants experimental design conducted in an ecologically-valid restaurant setting, we investigated the appreciation of differently-shaped pieces of cornmeal (rounded vs. rectangular) for different serving temperatures (room-temperature vs. hot presentation). 282 participants (165 female, mean age of 37 years, SD = 11 years, age range 17–62 years) tasted the cornmeal and evaluated it with respect to several sensorial qualities (i.e., temperature, wetness, texture, elasticity, and liking) and indicated the perceived dominant taste from amongst the five basic tastes. The results highlight that when tasted hot, cornmeal is evaluated as being significantly wetter, softer, as well as more elastic. As expected, irrespective of the actual serving temperature, the rounded form of presentation is preferred to the rectangular serving of cornmeal. The cornmeal was evaluated as tasting salty, with the umami taste also significantly attributed only to the rectangular cornmeal when served at room-temperature. A further control study conducted with a sample of 30 participants tasting pieces of irregularly-shaped room-temperature cornmeal further demonstrated the salty and umami taste of the cornmeal sampled in the present study. These results highlight the importance of serving temperature and how temperature interacts with food shape. The findings are discussed in the context of current theories of crossmodal correspondences, with consideration of the importance of temperature perception in gastronomic settings.

Marketing sonified fragrance: Designing soundscapes for scent.

Auditory branding is undoubtedly becoming more important across a range of sectors. One area, in particular, that has recently seen significant growth concerns the introduction of music and soundscapes that have been specifically designed to match a particular scent (what one might think of as "audio scents" or "sonic scents"). This represents an exciting new approach to the sensory marketing of fragrance and for industries with strategic sensory goals, such as cosmetics. Crucially, techniques such as the semantic differential technique, as well as the emerging literature on crossmodal correspondences, offer both a mechanistic understanding of, and a practical framework for, those wishing to rigorously align the connotative meaning and conceptual/emotional/sensory associations of sound and scent. These developments have enabled those working in the creative industries to start moving beyond previously popular approaches to matching, or translating between the senses, that were traditionally often based on the idiosyncratic phenomenon of synaesthesia, toward a more scientific approach while nevertheless still enabling/requiring a healthy dose of artistic inspiration. In this narrative historical review, we highlight the various approaches to the systematic matching of sound with scent and review the various marketing activations that have appeared in this space recently.

Prokofiev was (almost) right: A cross-cultural investigation of auditory-conceptual associations in Peter and the Wolf

Over recent decades, studies investigating cross-modal correspondences have documented the existence of a wide range of consistent cross-modal associations between simple auditory and visual stimuli or dimensions (e.g., pitch-lightness). Far fewer studies have investigated the association between complex and realistic auditory stimuli and visually presented concepts (e.g., musical excerpts-animals). Surprisingly, however, there is little evidence concerning the extent to which these associations are shared across cultures. To address this gap in the literature, two experiments using a set of stimuli based on Prokofiev’s symphonic fairy tale Peter and the Wolf are reported. In Experiment 1, 293 participants from several countries and with very different language backgrounds rated the association between the musical excerpts, images and words representing the story’s characters (namely, bird, duck, wolf, cat, and grandfather). The results revealed that participants tended to consistently associate the wolf and the bird with the corresponding musical excerpt, while the stimuli of other characters were not consistently matched across participants. Remarkably, neither the participants’ cultural background, nor their musical expertise affected the ratings. In Experiment 2, 104 participants were invited to rate each stimulus on eight emotional features. The results revealed that the emotional profiles associated with the music and with the concept of the wolf and the bird were perceived as more consistent between observers than the emotional profiles associated with the music and the concept of the duck, the cat, and the grandpa. Taken together, these findings therefore suggest that certain auditory-conceptual associations are perceived consistently across cultures and may be mediated by emotional associations.

Individual differences in sensitivity to taste-shape crossmodal correspondences

People generally associate curved and symmetrical shapes with sweetness, while associating angular and asymmetrical shapes with the other basic tastes (e.g., sour, bitter). However, these group-level taste-shape correspondences likely conceal important variation at an individual-level. We examined the extent to which individuals vary in their sensitivity to crossmodal correspondence between curvature and symmetry, on the one hand, and the five basic taste qualities (sweet, bitter, salty, sour, and umami), on the other. In Experiment 1, participants matched shapes (curved vs. angular, symmetrical vs. asymmetrical) and taste words. In Experiment 2, participants performed a similar task, though this time using actual tastants. Given that people differ in their hedonic experience of such shapes and tastes, we also measured participants’ liking for each taste and shape separately. The results replicate the general crossmodal correspondences between curved-sweet and symmetrical-sweet stimuli. Furthermore, participants tended to match sour and bitter tastes with angular and asymmetrical stimuli. However, these group-level taste-shape correspondences coexist alongside substantial variation at the level of the individual. While some participants consistently matched specific tastes with curved and symmetrical stimuli, others consistently matched these tastes with angular and asymmetrical stimuli, or else did not show these taste-shape correspondences. Liking for curved and symmetrical stimuli was higher than for angular and asymmetrical stimuli. However, participants also differed considerably in the extent to which these visual features affected their liking. Overall, our findings highlight the substantial individual differences that are associated with the degree to which people associate and like shapes and tastes.

Tailoring sweetness sensitivity cued by affective pictures

Plenty of previous studies have observed the cross-modal correspondence effect of vision and taste. In particular, affective pictures were shown to be associated with certain taste words. However, the extent to which such associations influence real taste perception remains unclear. The current study aims to assess the role of affective visual stimuli in the cross-modal modulation of sweet taste sensitivity. We utilized the same affective pictures from our previous study and categorized them into nine sub-groups based on different valence and arousal levels. Twenty volunteers (7 males and 13 females, average age 21.0 ± 0.7 years) were recruited to participate in the experiment. They were asked to taste various sucrose solutions after viewing each affective picture. Results showed that affective pictures' different valence and arousal levels could modify sweetness sensitivity. Specifically, sweetness sensitivity increases as the valence level changes from negative to positive. Positive pictures with high valence and intense arousal had the highest sweet sensitivity. On the other hand, as the arousal levels of negative pictures increased, sweetness sensitivity gradually decreased. Moreover, this visual-gustatory cross-modal integration was most significant around the sweetness threshold level. This study extends our understanding of the cross-modal interaction between vision and taste in a systematically tailored pattern. Specifically, affective visual stimuli have been shown to influence sweet taste sensitivity, with both valence and arousal levels playing essential roles. Such findings provide new insights into the complex interplay between sensory modalities and emotional states in shaping our taste perception.

Differences in attitude and purchase intention for multisensory stimuli in online stores with decreasing product congruence in terms of super additivity and cross-modal correspondence

Differences in attitude and purchase intention for multisensory stimuli in online stores with decreasing product congruence in terms of super additivity and cross-modal correspondence

Complexity Mediated Cross-modal Correspondence between Tone Sequences and Shapes

Complexity Mediated Cross-modal Correspondence between Tone Sequences and Shapes

Sonic Bouquet: Decoding Cross-Modal Correspondences Between Timbre and Scent

The examination of cross-modal correspondences between auditory and olfactory senses opens up an intriguing perspective into the study of extra-musical meaning. In a behavioral experiment, musically trained participants were presented with 26 complex synthetic tones and 12 aromatic stimuli. Their task was to report potential associations between the two. The data analysis revealed that the majority of scents featured at least one association with a sound that was above chance. The salient acoustical correlates of basic aromatic categories could be summarized as follows: both fruity (e.g., cherry, melon, and pomegranate) and sour aromas exhibited a positive correlation with pitch. Fruity scents, in addition, were more likely to be associated with sounds featuring pronounced low harmonic partials (1st–4th), low noise content, low roughness, and a greater number of distinct pitches. Conversely, sour aromas were linked with stronger energy in the higher frequencies. Sweet scents correlated with sounds characterized by a lower spectral centroid, whereas aromas in the spicy/other category were associated with weak lower partials (fundamental frequency in particular) and stronger noisy components.

Deep Boosting Learning: A Brand-new Cooperative Approach for Image-Text Matching.

Image-text matching remains a challenging task due to heterogeneous semantic diversity across modalities and insufficient distance separability within triplets. Different from previous approaches focusing on enhancing multi-modal representations or exploiting cross-modal correspondence for more accurate retrieval, in this paper we aim to leverage the knowledge transfer between peer branches in a boosting manner to seek a more powerful matching model. Specifically, we propose a brand-new Deep Boosting Learning (DBL) algorithm, where an anchor branch is first trained to provide insights into the data properties, with a target branch gaining more advanced knowledge to develop optimal features and distance metrics. Concretely, an anchor branch initially learns the absolute or relative distance between positive and negative pairs, providing a foundational understanding of the particular network and data distribution. Building upon this knowledge, a target branch is concurrently tasked with more adaptive margin constraints to further enlarge the relative distance between matched and unmatched samples. Extensive experiments validate that our DBL can achieve impressive and consistent improvements based on various recent state-of-the-art models in the image-text matching field, and outperform related popular cooperative strategies, e.g., Conventional Distillation, Mutual Learning, and Contrastive Learning. Beyond the above, we confirm that DBL can be seamlessly integrated into their training scenarios and achieve superior performance under the same computational costs, demonstrating the flexibility and broad applicability of our proposed method.

Cross-Modal Retrieval With Noisy Correspondence via Consistency Refining and Mining.

The success of existing cross-modal retrieval (CMR) methods heavily rely on the assumption that the annotated cross-modal correspondence is faultless. In practice, however, the correspondence of some pairs would be inevitably contaminated during data collection or annotation, thus leading to the so-called Noisy Correspondence (NC) problem. To alleviate the influence of NC, we propose a novel method termed Consistency REfining And Mining (CREAM) by revealing and exploiting the difference between correspondence and consistency. Specifically, the correspondence and the consistency only be coincident for true positive and true negative pairs, while being distinct for false positive and false negative pairs. Based on the observation, CREAM employs a collaborative learning paradigm to detect and rectify the correspondence of positives, and a negative mining approach to explore and utilize the consistency. Thanks to the consistency refining and mining strategy of CREAM, the overfitting on the false positives could be prevented and the consistency rooted in the false negatives could be exploited, thus leading to a robust CMR method. Extensive experiments verify the effectiveness of our method on three image-text benchmarks including Flickr30K, MS-COCO, and Conceptual Captions. Furthermore, we adopt our method into the graph matching task and the results demonstrate the robustness of our method against fine-grained NC problem. The code is available on https://github.com/XLearning-SCU/2024-TIP-CREAM.

Exploring the senses of taste with young children: Multisensory discoveries of food

Offering children multiple occasions and settings to approach new or least-liked foods has value both from a taste development perspective as well as a pedagogical one. New food experiences in positive atmospheres foster pleasure, curiosity and willingness to interact with these ingredients in multiple manners, not only eating or tasting them. This qualitative case study reports 4 and 5-year-old children’s articulation of their experiences in exploratory food ateliers at their preschool. During the study the children’s ideas and perceptions about their sensory experiences with food were documented, while keeping the children at the center of their explorations, in line with the Reggio Emilia approach. Results indicate that the children reflected on changes in their own attitudes and taste toward least-liked foods, as well as including meta-reflections on complex phenomena such as multisensory perception, cross-modal correspondences and taste development. Thus, by promoting multiple multisensory explorations, the study suggests that children can become more open-minded and self-aware, broadening the spectrum of food experiences.

Poster Session II: Kandinsky was right: Few do "express bright yellow in the bass notes, or dark lake in the treble".

Cross-modal correspondence is a sense of the inherent belongingness between two different senses; in our study these were pitch and color. Our goal was to investigate the confound in previous literature of individual differences in color brightness and pitch loudness. We tested twenty male participants. We determined equal brightness for each participant, across six colors: red, orange, yellow, green, blue, and purple; and equal loudness across seven pitches: 125, 250, 500, 2000, 4000, 8000, and 12,500 Hz. Then participants matched pitch with color in three different conditions: prototypical color hues, gray scale, and isobright colors. Our results indicated that in the prototypical condition, the participants chose yellow for high pitches, and blue and purple for the lowest pitches. In the gray scale condition, they chose white for high pitches and black for low pitches. These findings are consistent with previous research in the literature. However, we found that when controlling for individual differences in brightness, participants still chose yellow with higher pitches. Thus, there appears to be an inherent sense of belongingness between yellow and high pitches, even when controlling for the confounds of individual differences in brightness and loudness.

The Music of Perfume

Reported studies about crossmodal correspondences between music and smell basically focus on individual musical parameters. An experiment was carried out to explore such correspondences emerging from musical improvisation elicited by 20 olfactory stimuli, which allows the study of multiple musical parameters at the same time. A group of 14 pianists was asked to smell each stimulus and to play a short free improvisation inspired by it. From each improvisation, 14 musical parameters were extracted. The same odorants were also described by a panel of 15 volunteers. The main outcomes were the following: 1) The mean sensory ratings on a scale of fresh vs. warm appeared correlated with the average pitch of the improvisation. 2) The four odorants perceived as somewhat camphoraceous like lavender and mint yielded more non legato/staccato articulation or rests. 3) The feminine odor character was negatively correlated with the ambitus of the improvisation, defined as the difference between the highest and lowest note, and was positively correlated with pitch-class entropy. 4) Pleasantness yielded a negative correlation with pitch-class entropy and dissonance, being positively correlated with the lowest note. The first outcome is consistent with earlier studies, but outcomes 2–4 were novel findings.

Designing the rotary switch operation feeling based on cross-modal correspondences between tactile and auditory sensations

In recent years, the number of physical switches in in-vehicle devices has been reduced by consolidating several different control functions into a single control interface through active tactile devices. However, for such tactile devices to provide an improved user experience, the design must be carefully created to provide an appropriate operational feeling for each control function. The aim of this study was to find the best suitable combination of tactile and auditory sensations for the rotary knob-type tactile device to improve the user experience. The Semantic Differential method was used in the first experiment to evaluate electronically generated operating sounds, which enables the determination of the relationship between auditory impressions and acoustic features. In the second experiment, "the suitability of the combination of stimuli" was used to evaluate the impressions as an index during the simultaneous presentation of tactile and operating sounds. The results revealed a tendency for the stimuli to be perceived as "well-combined," with cross-modal correspondences between tactile and auditory sensations.

How many categories are there in crossmodal correspondences? A study based on exploratory factor analysis.

Humans naturally associate stimulus features of one sensory modality with those of other modalities, such as associating bright light with high-pitched tones. This phenomenon is called crossmodal correspondence and is found between various stimulus features, and has been suggested to be categorized into several types. However, it is not yet clear whether there are differences in the underlying mechanism between the different kinds of correspondences. This study used exploratory factor analysis to address this question. Through an online experiment platform, we asked Japanese adult participants (Experiment 1: N = 178, Experiment 2: N = 160) to rate the degree of correspondence between two auditory and five visual features. The results of two experiments revealed that two factors underlie the subjective judgments of the audiovisual crossmodal correspondences: One factor was composed of correspondences whose auditory and visual features can be expressed in common Japanese terms, such as the loudness-size and pitch-vertical position correspondences, and another factor was composed of correspondences whose features have no linguistic similarities, such as pitch-brightness and pitch-shape correspondences. These results confirm that there are at least two types of crossmodal correspondences that are likely to differ in terms of language mediation.

Reflections on Cross-Modal Correspondences: Current Understanding and Issues for Future Research.

The past two decades have seen an explosion of research on cross-modal correspondences. Broadly speaking, this term has been used to encompass associations between and among features, dimensions, or attributes across the senses. There has been an increasing interest in this topic amongst researchers from multiple fields (psychology, neuroscience, music, art, environmental design, etc.) and, importantly, an increasing breadth of the topic's scope. Here, this narrative review aims to reflect on what cross-modal correspondences are, where they come from, and what underlies them. We suggest that cross-modal correspondences are usefully conceived as relative associations between different actual or imagined sensory stimuli, many of these correspondences being shared by most people. A taxonomy of correspondences with four major kinds of associations (physiological, semantic, statistical, and affective) characterizes cross-modal correspondences. Sensory dimensions (quantity/quality) and sensory features (lower perceptual/higher cognitive) correspond in cross-modal correspondences. Cross-modal correspondences may be understood (or measured) from two complementary perspectives: the phenomenal view (perceptual experiences of subjective matching) and the behavioural response view (observable patterns of behavioural response to multiple sensory stimuli). Importantly, we reflect on remaining questions and standing issues that need to be addressed in order to develop an explanatory framework for cross-modal correspondences. Future research needs (a) to understand better when (and why) phenomenal and behavioural measures are coincidental and when they are not, and, ideally, (b) to determine whether different kinds of cross-modal correspondence (quantity/quality, lower perceptual/higher cognitive) rely on the same or different mechanisms.