Crossmodal Correspondences Research Articles

Crossmodal Correspondence between Music and Ambient Color Is Mediated by Emotion.

The quality of a concert hall primarily depends on its acoustics. But does visual input also have an impact on musical enjoyment? Does the color of ambient lighting modulate the perceived music quality? And are certain colors perceived to fit better than others with a given music piece? To address these questions, we performed three within-subjects experiments. We carried out two pretests to select four music pieces differing in tonality and genre, and 14 lighting conditions of varying hue, brightness, and saturation. In the main experiment, we applied a fully crossed repeated-measures design. Under each of the four lighting conditions, participants rated the musical variables 'Harmonic', 'Powerful', 'Gloomy', 'Lively' and overall liking of the music pieces, as well as the perceived fit of music and lighting. Subsequently, participants evaluated music and lighting separately by rating the same variables as before, as well as their emotional impact (valence, arousal, dominance). We found that music and lighting being similarly rated in terms of valence and arousal in the unimodal conditions were judged to match better when presented together. Accordingly, tonal (atonal) music was rated to fit better with weakly saturated (highly saturated) colors. Moreover, some characteristics of the lighting were carried over to music. That is, just as red lighting was rated as more powerful than green and blue lighting, music was evaluated to be more powerful under red compared to green and blue lighting. We conclude that listening to music is a multisensory process enriched by impressions from the visual domain.

Crossmodal Congruency Between Background Music and the Online Store Environment: The Moderating Role of Shopping Goals.

Despite the robust evidence that congruent background music in the physical store environment positively affects consumer reactions, less is known about its effects in an online context. The present study aims (1) to examine whether congruency via multiple elicited crossmodal correspondences between background music and the online store environment (e.g., perceived lightness, loudness, and coldness of the cue/environment) leads to more positive affective, evaluative, and behavioral consumer reactions and (2) to investigate the moderating role of shopping goals on this crossmodal congruency effect. Previous research showed that low task-relevant atmospheric cues like music can have a negative effect on consumers when they visit a website with a purchase goal in mind. An online experiment was conducted with 239 respondents randomly assigned to a shopping goal (experiential browsing vs. goal-directed searching) and a music condition (no music, crossmodally congruent music, or crossmodally incongruent music). Our results show that crossmodally incongruent background music (vs. no music) leads to more positive consumer reactions for experiential browsers and more negative consumer reactions for goal-directed searchers. Conversely, crossmodally congruent background music (vs. no music) has a positive effect on experiential browsers and no adverse effect on goal-directed searchers. Additionally, the presence of crossmodally congruent background music leads to more positive consumer reactions than the presence of crossmodally incongruent background music, independent of the shopping goal. We extend previous research on multisensory congruency effects by showing the added value of establishing congruency between music and the store environment via multiple elicited crossmodal correspondences in the online environment, countering previously found negative effects of low-task relevant atmospheric cues for goal-directed searchers.

Tasting prosody: Crossmodal correspondences between voice quality and basic tastes

Voice quality or type of phonation (e.g., a whispery voice) can prime specific sensory associations amongst consumers. In the realm of sensory and consumer science, a wide range of taste-sound correspondences has been documented. A growing body of research on crossmodal correspondences has revealed that people reliably associate sounds with basic taste qualities. Here, we examined the largely unexplored associations between basic tastes and sounds: namely taste-voice quality correspondences. Across three pre-registered studies, participants associated four types of voice qualities (modal, whispery, creaky, and falsetto) with the five basic tastes (sweet, sour, salty, bitter, and umami).Study 1 investigated the relations between voice qualities and taste words. Study 2 attempted to replicate the findings and revealed the underpinning psychological mechanisms in terms of semantic/emotional associations. Study 3 used the descriptions of food products that varied in terms of their taste to expand the applicability of the findings. The results demonstrated that participants reliably associate specific voice qualities with particular tastes. Falsetto voices are matched more strongly with sweetness than other voices. Creaky voices are matched more strongly with bitterness than other voices. Modal voices are matched more strongly with umami than creaky voices. Evaluation/positive valence might partially underlie the associations between sweet/bitter-voice quality correspondences. Taken together, these findings reveal a novel case of sound-taste correspondences and deepen our understanding of how people can associate attributes from different senses.

Spontaneous crossmodal correspondences grounded in contexts

Thoughts and associations are spontaneously generated and situated. A growing body of research on crossmodal correspondences has revealed that individuals tend to associate information across sensory modalities. However, most of the findings have been based on restricted sensory pairs/items and the role of context remains largely unexplored. Relying on crossmodal correspondence studies and the theory of situated conceptualization, the present study aimed to study spontaneous crossmodal correspondences and the role of context (background settings, culture) in modulating them. Japanese and British participants (n = 604) were presented with shape stimuli (round and angular) and spontaneously generated words describing the stimuli in different contexts (geometric shapes, logo in food packaging, and logo in non-food products). The results demonstrated that individuals spontaneously associate shape features with different senses, something that is influenced by context (background settings, culture). Geometric shapes were associated with visual and auditory descriptors, while shapes in the food context were linked with chemosensory (olfactory and taste) descriptors. Moreover, Japanese participants were more likely to generate touch descriptors, while British participants were more likely to use auditory descriptors. Together, our findings reveal how multisensory associations are spontaneously generated and how these associations are grounded in contexts.

Pitch-Luminance Crossmodal Correspondence in the Baby Chick: An Investigation on Predisposed and Learned Processes.

Our senses are constantly reached by a multitude of stimuli from all different sensory modalities. To create a coherent representation of the environment, we must integrate the various unimodal inputs that refer to the same object into a single multimodal representation. In some cases, however, we tend to bind certain properties of the stimuli without any apparent reason, which is a phenomenon named crossmodal correspondence. For instance, we match a spiky or a rounded shape with the sound “Kiki” or “Bouba”, respectively. Similarly, we associate the left hemispace with low luminance and the right one with high luminance. Instances of crossmodal correspondences were described also in other mammals, and recently, a case of space-luminance crossmodal correspondence was reported in birds (i.e., domestic chicks). Here, we investigate the presence of pitch–luminance crossmodal correspondence in three-day-old chicks, employing experimental methods that exploit either predisposed or learned processes. While failing to report evidence for this phenomenon, we discuss the difference between statistical and structural crossmodal correspondences and the possible role of environmental factors in determining their emergence. Moreover, we discuss the importance of the different experimental methodologies to investigate distinct aspects of this perceptual phenomenon to reach a deeper understanding and unveil the role of innate vs. learned mechanisms.

The taste of visual textures

Numerous crossmodal correspondences between visual elements and basic tastes have been documented in recent years. Research has shown that many of these correspondences can influence taste expectations when applied in food packaging. However, research on correspondences between visual textures and tastes is scarce, despite the ability of the former to convey specific information about materials and objects. In the present study, we conducted two online experiments designed to study crossmodal correspondences between basic taste words and visual textures with common material properties. In Experiment 1 (N = 194), we evaluated explicit associations between six visual texture categories (with four levels of each category) and basic taste words. The results revealed moderate associations between one of the fluffy visual textures and sweetness and between a rough and a crunchy visual texture and saltiness. In Experiment 2 (N = 407), we superimposed the visual textures associated with the basic tastes found in Experiment 1 on food extrinsic factors (i.e., packaging, napkin) served in combination with products of three taste qualities (i.e., neutral/ambiguous, sweet, salty). We did not find evidence supporting the idea that visual textures that are crossmodally corresponding to specific tastes, as revealed in Experiment 1, influenced taste expectations. The results of the study suggest that the strength of the crossmodal correspondence between visual textures and basic taste words studied here is moderate.

Predicting the crossmodal correspondences of odors using an electronic nose

When designing multisensorial experiences, robustly predicting the crossmodal perception of olfactory stimuli is a critical factor. We investigate the possibility of predicting olfactory crossmodal correspondences using the underlying physicochemical features. An electronic nose was tuned to the crossmodal perceptual axis of olfaction and was used to foretell people's crossmodal correspondences between odors and the angularity of shapes, smoothness of texture, perceived pleasantness, pitch, and colors. We found that the underlying physicochemical features of odors could be used to predict people's crossmodal correspondences. The human-machine perceptual dimensions that correlated well are the angularity of shapes (r = 0.71), the smoothness of texture (r = 0.82), pitch (r = 0.70), and the lightness of color (r = 0.59). The human-machine perceptual dimensions that did not correlate well (r < 0.50) are the perceived pleasantness (r = 0.20) and the hue of the color (r = 0.42 & 0.44). All perceptual dimensions except for the perceived pleasantness could be robustly predicted (p-values < 0.0001) including the hue of color. While it is recognized that olfactory perception is strongly shaped by learning and experience, our findings suggest that there is a systematic and predictable link between the physicochemical features of odorous stimuli and crossmodal correspondences. These findings may provide a crucial building block towards the digital transmission of smell and enhancing multisensorial experiences with better designs as well as more engaging, and enriched experiences.

Synesthesia in a congenitally blind individual

Synesthesia represents an atypical merging of percepts, in which a given sensory experience (e.g., words, letters, music) triggers sensations in a different perceptual domain (e.g., color). According to recent estimates, the vast majority of the reported cases of synesthesia involve a visual experience. Purely non-visual synesthesia is extremely rare and to date there is no reported case of a congenitally blind synesthete. Moreover, it has been suggested that congenital blindness impairs the emergence of synesthesia-related phenomena such as multisensory integration and cross-modal correspondences between non-visual senses (e.g., sound-touch). Is visual experience necessary to develop synesthesia? Here we describe the case of a congenital blind man (CB) reporting a complex synesthetic experience, involving numbers, letters, months and days of the week. Each item is associated with a precise position in mental space and with a precise tactile texture. In one experiment we empirically verified the presence of number-texture and letter-texture synesthesia in CB, compared to non-synesthete controls, probing the consistency of item-texture associations across time and demonstrating that synesthesia can develop without vision. Our data fill an important void in the current knowledge on synesthesia and shed light on the mechanisms behind sensory crosstalk in the human mind.

Cross-modal Graph Matching Network for Image-text Retrieval

Image-text retrieval is a fundamental cross-modal task whose main idea is to learn image-text matching. Generally, according to whether there exist interactions during the retrieval process, existing image-text retrieval methods can be classified into independent representation matching methods and cross-interaction matching methods. The independent representation matching methods generate the embeddings of images and sentences independently and thus are convenient for retrieval with hand-crafted matching measures (e.g., cosine or Euclidean distance). As to the cross-interaction matching methods, they achieve improvement by introducing the interaction-based networks for inter-relation reasoning, yet suffer the low retrieval efficiency. This article aims to develop a method that takes the advantages of cross-modal inter-relation reasoning of cross-interaction methods while being as efficient as the independent methods. To this end, we propose a graph-based Cross-modal Graph Matching Network (CGMN) , which explores both intra- and inter-relations without introducing network interaction. In CGMN, graphs are used for both visual and textual representation to achieve intra-relation reasoning across regions and words, respectively. Furthermore, we propose a novel graph node matching loss to learn fine-grained cross-modal correspondence and to achieve inter-relation reasoning. Experiments on benchmark datasets MS-COCO, Flickr8K, and Flickr30K show that CGMN outperforms state-of-the-art methods in image retrieval. Moreover, CGMM is much more efficient than state-of-the-art methods using interactive matching. The code is available at https://github.com/cyh-sj/CGMN .

Seeing Sounds: The Role of Vowels and Consonants in Crossmodal Correspondences.

Crossmodal correspondences refer to the fact that certain domains of features in different sensory modalities are associated with each other. Here, we investigated the crossmodal correspondences between speech sounds and visual shapes. Specifically, we tested whether the classification dimensions of English vowels (front–central–back) and consonants (voiced–voiceless, sonorant–obstruent, and stop–continuant) correspond to visual shapes along a bipolar rounded–angular dimension. We adapted eighteen meaningless pseudowords from a previous study that corresponded to either the round or the sharp concept. On each trial, the participants heard one of the pseudowords and saw a rounded shape and an angular shape presented side-by-side on the monitor. Participants judged which shape provided a better match to the spoken pseudoword. A logistic regression was conducted in order to elucidate the effectiveness of classification dimensions of phonemes when predicting variations in the sound–shape matchings. The results demonstrated that the sound–shape matchings were predictable using front–central–back dimensions of vowels, and voiced–voiceless and stop–continuant dimensions of consonants. Hence, we verified that sound–shape matching is underpinned by contrasting dimensions in both vowels and consonants, therefore demonstrating crossmodal correspondences at the phonetic level.

Pitch-elevation and pitch-size cross-modal correspondences do not affect temporal ventriloquism.

Temporal ventriloquism refers to the shift in the perceived timing of a visual stimulus towards a transient auditory stimulus presented close in time. This effect is demonstrated by greater sensitivity of temporal order judgments of two visual stimuli when a sound is presented before the first visual stimulus and after the second visual stimulus. Recent studies suggest that temporal ventriloquism is affected by cross-modal correspondence between auditory pitch and visual elevation but not by correspondence between pitch and visual size. Here we examined the possibility that these results do not reflect a difference in the effects of different types of cross-modal correspondences on temporal ventriloquism but are rather mediated by shifts in visual-spatial attention triggered by preceding auditory stimuli. In Experiment 1, we replicated the results of previous studies that varied with the type of correspondence. Experiment 2 investigated the effects of the second audiovisual stimuli's asynchrony while the first audiovisual stimuli were synchronized. The results, unlike in Experiment 1, revealed that the magnitude of the temporal ventriloquism effect did not change with the congruency of pitch-elevation correspondence. Experiment 3 also indicated that the asynchrony of the first audiovisual stimuli modulated visual discrimination sensitivity irrespective of temporal ventriloquism. These results suggest that cross-modal correspondences do not affect temporal ventriloquism. Greater visual sensitivity when audiovisual stimuli are congruent with pitch-elevation correspondence may be attributable to shifts in visual attention caused by pitches of the preceding auditory stimulus, which speeds up detection of the first visual stimulus.

Physicochemical features partially explain olfactory crossmodal correspondences

Physicochemical features partially explain olfactory crossmodal correspondences

That sounds healthy! Audio and visual frequency differences in brand sound logos modify the perception of food healthfulness

Sonic brand logos, also termed “sogos”, are a marketing communication tool that brands use to signify brand or product benefits to consumers in catchy, non-visual ways. Given the considerable utility of brand lsogos, it is surprising that there has been scant research into the nature of the specific acoustic features that can be modulated to connote certain traits, such as the healthfulness of products within the food category. Our findings revealed that sogos created with higher (vs. lower) frequency were significantly matched with healthy food products (vs. less healthy), while the effect of tempo was neutral. This effect generalizes to high (vs. low) spatial frequency visual stimuli too. The current study contributes to the literature on the crossmodal correspondence between acoustic sound clips and expectations of healthfulness. It also advances the theoretical insights into business applications using optimal sogos congruent with visual cues on packaging to connote food healthfulness to consumers implicitly.

High-pitch sounds small for domestic dogs: abstract crossmodal correspondences between auditory pitch and visual size.

Humans possess intuitive associations linking certain non-redundant features of stimuli—e.g. high-pitched sounds with small object size (or similarly, low-pitched sounds with large object size). This phenomenon, known as crossmodal correspondence, has been identified in humans across multiple different senses. There is some evidence that non-human animals also form crossmodal correspondences, but the known examples are mostly limited to the associations between the pitch of vocalizations and the size of callers. To investigate whether domestic dogs, like humans, show abstract pitch-size association, we first trained dogs to approach and touch an object after hearing a sound emanating from it. Subsequently, we repeated the task but presented dogs with two objects differing in size, only one of which was playing a sound. The sound was either high or low pitched, thereby creating trials that were either congruent (high pitch from small object; low pitch from large objects) or incongruent (the reverse). We found that dogs reacted faster on congruent versus incongruent trials. Moreover, their accuracy was at chance on incongruent trials, but significantly above chance for congruent trials. Our results suggest that non-human animals show abstract pitch sound correspondences, indicating these correspondences may not be uniquely human but rather a sensory processing feature shared by other species.

Analogical mapping across sensory modalities and evidence for a general analogy factor

Analogy is a central component of human cognition. Analogical “mapping” of similarities between pieces of information present in our experiences supports cognitive and social development, classroom learning, and creative insights and innovation. To date, analogical mapping has primarily been studied within separate modalities of information (e.g., verbal analogies between words, visuo-spatial analogies between objects). However, human experience, in development and adulthood, includes highly variegated information (e.g., words, sounds, objects) received via multiple sensory and information-processing pathways (e.g., visual vs. auditory pathways). Whereas cross-modal correspondences (e.g., between pitch and height) have been observed, the correspondences were between individual items, rather than between relations. Thus, analogical mapping (characterized by second-order relations between relations) has not been directly tested as a basis for cross-modal correspondence. Here, we devised novel cross-modality analogical stimuli (lines-to-sounds, lines-to-words, words-to-sounds) that explicated second-order comparisons between relations. In four samples across three studies-participants demonstrated well-above-chance identification of cross-modal second-order relations, providing robust evidence of analogy across modalities. Further, performance across all analogy types was explained by a single factor, indicating a modality-general analogical ability (i.e., an “analo-g” factor). Analo-g explained performance over-and-above fluid intelligence as well as verbal and spatial abilities, though a stronger relationship to verbal than visuo-spatial ability emerged, consistent with verbal/semantic contributions to analogy. The present data suggests novel questions about our ability to find/learn second-order relations among the diverse information sources that populate human experience, and about cross-modal human and AI analogical mapping in developmental, educational, and creative contexts.

Cross-modal correspondence between visual information and taste perception of bitter foods and drinks

People are sensitive to bitter taste because bitterness serves as a warning sign of poisons. Previous research reported that taste receptors in the oral cavity are more sensitive to bitter taste than to sweet taste. Other sensory systems, i.e., auditory, olfactory, tactile and visual, have also been shown to be involved in the sense of taste, a phenomenon known as cross-modal correspondence. It has been shown that the color of food or its packaging influences taste expectation and perception. However, it remains unknown whether cross-modal correspondence between visual information and taste depends on the degree of bitterness. Here, we investigated the effect of background color on taste using chocolate and green tea with varying bitterness levels. Undergraduate students were asked to evaluate how bitter and how sweet milk chocolate (Experiment 1) or less bitter and bitter chocolate (Experiment 2) tasted when served from black or pink wrapping paper. In Experiment 3, undergraduate students were ask to evaluate how bitter and how sweet less bitter and bitter green tea tasted when served from a clear blue or a clear cup. The experiments showed that when chocolate and green tea were bitter, visual information influenced taste evaluation. In contrast, no visual effect on taste was seen for less bitter chocolate, milk chocolate or less bitter green tea. The results suggest the possibility that cross-modal correspondence of visual information to taste changes depending on the degree of bitterness.

Differences in attitude and purchase intention for multisensory stimuli in online stores with decreasing product congruence in terms of superadditivity and cross-modal correspondence

Differences in attitude and purchase intention for multisensory stimuli in online stores with decreasing product congruence in terms of superadditivity and cross-modal correspondence

Smelling sensations: Olfactory crossmodal correspondences (JPI-first)

Olfaction is ingrained into the fabric of our daily lives and constitutes an integral part of our perceptual reality. Within this reality, there are crossmodal interactions and sensory expectations; understanding how olfaction interacts with other sensory modalities is crucial for augmenting interactive experiences with more advanced multisensorial capabilities. This knowledge will eventually lead to better designs, more engaging experiences, and enhancing the perceived quality of experience. Toward this end, the authors investigated a range of crossmodal correspondences between ten olfactory stimuli and different modalities (angularity of shapes, smoothness of texture, pleasantness, pitch, colors, musical genres, and emotional dimensions) using a sample of 68 observers. Consistent crossmodal correspondences were obtained in all cases, including our novel modality (the smoothness of texture). These associations are most likely mediated by both the knowledge of an odor’s identity and the underlying hedonic ratings: the knowledge of an odor’s identity plays a role when judging the emotional and musical dimensions but not for the angularity of shapes, smoothness of texture, perceived pleasantness, or pitch. Overall, hedonics was the most dominant mediator of crossmodal correspondences.

Crossmodal Correspondence Between Auditory Timbre and Visual Shape.

Crossmodal correspondences are defined as associations between crossmodal stimuli based on seemingly irrelevant stimulus features (i.e., bright shapes being associated with high-pitched sounds). There is a large body of research describing auditory crossmodal correspondences involving pitch and volume, but not so much involving auditory timbre, the character or quality of a sound. Adeli and colleagues (2014, Front. Hum. Neurosci. 8, 352) found evidence of correspondences between timbre and visual shape. The present study aimed to replicate Adeli et al.'s findings, as well as identify novel timbre-shape correspondences. Participants were tested using two computerized tasks: an association task, which involved matching shapes to presented sounds based on best perceived fit, and a semantic task, which involved rating shapes and sounds on a number of scales. The analysis of association matches reveals nonrandom selection, with certain stimulus pairs being selected at a much higher frequency. The harsh/jagged and smooth/soft correspondences observed by Adeli et al. were found to be associated with a high level of consistency. Additionally, high matching frequency of sounds with unstudied timbre characteristics suggests the existence of novel correspondences. Finally, the ability of the semantic task to supplement existing crossmodal correspondence assessments was demonstrated. Convergent analysis of the semantic and association data demonstrates that the two datasets are significantly correlated (-0.36) meaning stimulus pairs associated with a high level of consensus were more likely to hold similar perceived meaning. The results of this study are discussed in both theoretical and applied contexts.

The Effects of Mandarin Chinese Lexical Tones in Sound-Shape and Sound-Size Correspondences.

Crossmodal correspondences refer to when specific domains of features in different sensory modalities are mapped. We investigated how vowels and lexical tones drive sound-shape (rounded or angular) and sound-size (large or small) mappings among native Mandarin Chinese speakers. We used three vowels (/i/, /u/, and /a/), and each vowel was articulated in four lexical tones. In the sound-shape matching, the tendency to match the rounded shape was decreased in the following order: /u/, /i/, and /a/. Tone 2 was more likely to be matched to the rounded pattern, whereas Tone 4 was more likely to be matched to the angular pattern. In the sound-size matching, /a/ was matched to the larger object more than /u/ and /i/, and Tone 2 and Tone 4 correspond to the large-small contrast. The results demonstrated that both vowels and tones play prominent roles in crossmodal correspondences, and sound-shape and sound-size mappings are heterogeneous phenomena.