Crossmodal Correspondences Research Articles

Does cross-modal correspondence modulate modality-specific perceptual processing? Study using timing judgment tasks.

Cross-modal correspondences refer to associations between stimulus features across sensory modalities. Previous studies have shown that cross-modal correspondences modulate reaction times for detecting and identifying stimuli in one modality when uninformative stimuli from another modality are present. However, it is unclear whether such modulation reflects changes in modality-specific perceptual processing. We used two psychophysical timing judgment tasks to examine the effects of audiovisual correspondences on visual perceptual processing. In Experiment 1, we conducted a temporal order judgment (TOJ) task that asked participants to judge which of two visual stimuli presented with various stimulus onset asynchronies (SOAs) appeared first. In Experiment 2, we conducted a simultaneous judgment (SJ) task that asked participants to report whether the two visual stimuli were simultaneous or successive. We also presented an unrelated auditory stimulus, simultaneously or preceding the first visual stimulus, and manipulated the congruency between audiovisual stimuli. Experiment 1 indicated that the points of subjective simultaneity (PSSs) between the two visual stimuli estimated in the TOJ task shifted according to the audiovisual correspondence between the auditory pitch and visual features of vertical location and size. However, these audiovisual correspondences did not affect PSS estimated using the SJ task in Experiment 2. The different results of the two tasks can be explained through the response bias triggered by audiovisual correspondence that only the TOJ task included. We concluded that audiovisual correspondence would not modulate visual perceptual timing and that changes in modality-specific perceptual processing might not trigger the congruency effects reported in previous studies.

Tasty vibes: Uncovering crossmodal correspondences between tactile vibrations and basic tastes

The interest in crossmodal correspondences individually involving the senses of touch and taste has grown rapidly in the last few decades. Several correspondences involving different tactile dimensions (e.g., hardness/softness, roughness/smoothness) have been uncovered, such as those between sweetness and softness and between roughness and sourness. However, a dimension that has been long overlooked, despite its pervasiveness and importance in everyday experiences, relates to tactile vibrations. The present study aimed to fill this gap and investigate crossmodal correspondences between basic tastes and vibrations. In the present study (N = 72), we uncovered these associations by having participants sample basic taste (i.e., sweet, salty, sour, bitter, umami) aqueous solutions and chose the frequency of vibrations, delivered via a consumer-grade subwoofer wristband on their dominant hand, that they most strongly associated with each taste. We found that sourness was most strongly associated with frequencies around 98 Hz, and that sweetness and umami were associated with frequencies around 77 Hz. These correspondences may, to different extents, be based on affective and semantic mechanisms. The findings have relevant implications for theoretical research on multisensory integration and perception and the potential future applications of these associations, through wearable technologies, to enhance eating experiences and promote healthier eating habits.

The taste of colours

A multitude of crossmodal correspondences have now been documented between taste (gustation) and visual features (such as hue). In the present study, new analytical methods are used to investigate taste-colour correspondences in a more fine-grained manner while also investigating potential underlying mechanisms. In Experiment 1, image processing analysis is used to evaluate whether searching online for visual images associated with specific taste words (e.g., bitter, sweet) generates outcomes with colour proportions similar to those that have been documented in the literature on taste–colour correspondences. Colour–taste matching tasks incorporating a much wider colour space than tested in previous studies, were assessed in Experiments 2 and 3. Experiments 3 and 4 assessed the extent to which the statistical regularities of the environment, as captured by food object categories, might help to explain the aforementioned correspondences and to what extent the correspondences are present in online content associated with specific tastes, respectively. Experiment 5 evaluated the role of statistical regularities in underpinning colour-taste correspondences related to the stage of ripening of fruit. Overall, the findings revealed consistent associations between specific colours and tastes, in a more nuanced manner than demonstrated in previous studies, while at the same time also showing that both food object categories and the stage of fruit ripening significantly influenced colour and taste perceptions. This, in turn, suggests that people might base these correspondences on both the foods present in their environments, as well as the natural changes that they undergo as they ripen. The results are discussed in light of the different accounts that have been suggested to explain colour-taste correspondences.

Odors modulate color appearance.

Our brain constantly combines multisensory information from our surrounding environment. Odors for instance are often perceived with visual cues; these sensations interact to form our own subjective experience. This integration process can have a profound impact on the resulting experience and can alter our subjective reality. Crossmodal correspondences are the consistent associations between stimulus features in different sensory modalities. These correspondences are presumed to be bidirectional in nature and have been shown to influence our perception in a variety of different sensory modalities. Vision is dominant in our multisensory perception and can influence how we perceive information in our other senses, including olfaction. We explored the effect that different odors have on human color perception by presenting olfactory stimuli while asking observers to adjust a color patch to be devoid of hue (neutral gray task). We found a shift in the perceived neutral gray point to be biased toward warmer colors. Four out of five of our odors also trend toward their expected crossmodal correspondences. For instance, when asking observers to perform the neutral gray task while presenting the smell of cherry, the perceptually achromatic stimulus was biased toward a red-brown. Using an achromatic adjustment task, we were able to demonstrate a small but systematic effect of the presence of odors on human color perception.

Development and experience-dependence of multisensory spatial processing.

Multisensory spatial processes are fundamental for efficient interaction with the world. They include not only the integration of spatial cues across sensory modalities, but also the adjustment or recalibration of spatial representations to changing cue reliabilities, crossmodal correspondences, and causal structures. Yet how multisensory spatial functions emerge during ontogeny is poorly understood. New results suggest that temporal synchrony and enhanced multisensory associative learning capabilities first guide causal inference and initiate early coarse multisensory integration capabilities. These multisensory percepts are crucial for the alignment of spatial maps across sensory systems, and are used to derive more stable biases for adult crossmodal recalibration. The refinement of multisensory spatial integration with increasing age is further promoted by the inclusion of higher-order knowledge.

Distinguishing between straight and curved sounds: Auditory shape in pitch, loudness, and tempo gestures.

Sound-based trajectories or sound gestures draw links to spatiokinetic processes. For instance, a gliding, decreasing pitch conveys an analogous downward motion or fall. Whereas the gesture's pitch orientation and range convey its meaning and magnitude, respectively, the way in which pitch changes over time can be conceived of as gesture shape, which to date has rarely been studied in isolation. This article reports on an experiment that studied the perception of shape in uni-directional pitch, loudness, and tempo gestures, each assessed for four physical scalings. Gestures could increase or decrease over time and comprised different frequency and sound level ranges, durations, and different scaling contexts. Using a crossmodal-matching task, participants could reliably distinguish between pitch and loudness gestures and relate them to analogous visual line segments. Scalings based on equivalent-rectangular bandwidth (ERB) rate for pitch and raw signal amplitude for loudness were matched closest to a straight line, whereas other scalings led to perceptions of exponential or logarithmic curvatures. The investigated tempo gestures, by contrast, did not yield reliable differences. The reliable, robust perception of gesture shape for pitch and loudness has implications on various sound-design applications, especially those cases that rely on crossmodal mappings, e.g., visual analysis or control interfaces like audio waveforms or spectrograms. Given its perceptual relevance, auditory shape appears to be an integral part of sound gestures, while illustrating how crossmodal correspondences can underpin auditory perception.

Harmony of senses: Exploring the impact of sound aesthetic features' on taste imagery

People reliably associate visual aesthetic features such as curvature and symmetry with tastes. In the present study, considering the transitive hypothesis of crossmodal correspondences, we evaluated whether these findings would extend to the relationship between sound aesthetic features and tastes, and whether feature-based congruency or affective priming would explain the influence of melodies on taste imagery. In Experiment 1, we evaluated how people associate different melody profiles (balanced vs. unbalanced, smooth vs. jagged, symmetrical vs. asymmetrical, simpler vs. more complex) with different tastes (sweet, sour, bitter, salty), as well as the melodies' associated fluency, valence, and arousal. Smooth and complex melodies were perceived as sweeter, jagged and unbalanced melodies as sourer, asymmetrical and jagged melodies as more bitter, and jagged and balanced melodies as saltier. In Experiment 2, we selected the most strongly associated aesthetic sound dimension with tastes, namely contour, and evaluated whether crossmodal congruency, based on crossmodal correspondences, or affective priming would influence people’s sensory and hedonic imagery associated with sweet and sour foods. In the imagery tasks, the participants showed higher sour ratings in the sourness task and higher sweetness ratings in the sweetness task. In the sweet imagery task, smooth melodies led to sweeter and less sour food imagery, whereas jagged melodies in the sourness task led to more sour and less sweet food imagery. These results favour the crossmodal congruency explanation rather than the affective alternative, which we ponder in the general discussion.

Marble melancholy: using crossmodal correspondences of shapes, materials, and music to predict music-induced emotions.

Music is known to elicit strong emotions in listeners, and, if primed appropriately, can give rise to specific and observable crossmodal correspondences. This study aimed to assess two primary objectives: (1) identifying crossmodal correspondences emerging from music-induced emotions, and (2) examining the predictability of music-induced emotions based on the association of music with visual shapes and materials. To achieve this, 176 participants were asked to associate visual shapes and materials with the emotion classes of the Geneva Music-Induced Affect Checklist scale (GEMIAC) elicited by a set of musical excerpts in an online experiment. Our findings reveal that music-induced emotions and their underlying core affect (i.e., valence and arousal) can be accurately predicted by the joint information of musical excerpt and features of visual shapes and materials associated with these music-induced emotions. Interestingly, valence and arousal induced by music have higher predictability than discrete GEMIAC emotions. These results demonstrate the relevance of crossmodal correspondences in studying music-induced emotions. The potential applications of these findings in the fields of sensory interactions design, multisensory experiences and art, as well as digital and sensory marketing are briefly discussed.

Perceptual Similarity: Insights From Crossmodal Correspondences

AbstractPerceptual similarity is one of the most fiercely debated topics in the philosophy and psychology of perception. The documented history of the issue spans all the way from Plato – who regarded similarity as a key factor for human perceptual experience and cognition – through to contemporary psychologists – who have tried to determine whether, and if so, how similarity relationships can be established between stimuli both within and across the senses. Recent research on cross-sensory associations, otherwise known as crossmodal correspondences – that is, the existence of observable consensual associations, or mappings, between stimuli across different senses – represents an especially interesting field in which to study perceptual similarity. In fact, most accounts of crossmodal association that have been put forward in the literature to date evoke perceptual similarity as a key explanatory factor mediating the underlying association. At the same time, however, these various accounts raise several important theoretical questions concerning the very nature of similarity, with, for example, the sensory, affective, or cognitive underpinnings of similarity judgements remaining unclear. We attempt to shed light on these questions by examining the various accounts of crossmodal associations that have been put forward in the literature. Our suggestion is that perceptual similarity varies from being phenomenologically-based to conceptually-based. In particular, we propose that the nature of the associations underlying similarity judgements – whether these associations are phenomenologically-, structurally-, emotionally-, or conceptually-based – may be represented in a two-dimensional space with associative strength on one axis, and cognitive penetrability on the other.

Electrophysiological evidence of crossmodal correspondence between auditory pitch and visual elevation affecting inhibition of return

Inhibition of return (IOR) has proved to be weakened by audiovisual integration because of the increased perceptual salience of targets. Although other audiovisual interactions, such as crossmodal correspondence, have also been shown to facilitate attentional processes, to the best of our knowledge, no study has investigated the interaction between crossmodal correspondence and IOR. The present study employed Posner’s spatial cueing paradigm and manipulated the cue validity, crossmodal correspondence congruency and time interval of auditory and visual stimuli (AV interval) to explore the effect of crossmodal correspondence on the IOR effect. The behavioral results showed a reduced IOR effect under the correspondence congruency condition in contrast to the correspondence incongruency condition at the AV interval of 200 ms, whereas at an AV interval of 80 ms, the decreased IOR effect under crossmodal correspondence congruency was eliminated. The electrophysiological results showed a reduced amplitude difference in P2 between valid and invalid cue conditions when the crossmodal correspondence effect decreased the IOR effect. The present study provided the first evidence of the weakened effect of the crossmodal correspondence effect on the IOR effect, which could be eliminated by audiovisual integration.

Crossmodal correspondences between visual features and tastes in preschoolers: an exploratory study.

Adults possess a natural inclination to associate sensory cues derived from distinct modalities, such as the pairing of sweet with pink. However, studies exploring crossmodal correspondences in children, particularly in the sensory pairing of visual features and tastes, are scant, leaving unanswered questions regarding the developmental trajectory of crossmodal correspondences. The present study investigates whether Japanese preschool children demonstrate specific biases in shape-color, shape-taste, and color-taste associations. In a series of in-person experiments, 92 children between 3 to 6 years of age completed matching tasks utilizing paper stimuli. Children exhibit crossmodal correspondences in shape-color (circle-red and asymmetrical star-yellow), shape-taste (triangle-salty and circle-sweet), and color-taste (yellow-sour, black-bitter, and pink-sweet) associations. Moreover, children's choices are not influenced by their individual preferences. The crossmodal correspondences observed in this study have been observed in previous research on adults from the same (Japanese) culture, although adults showed more crossmodal correspondences than the children in this study (e.g., pink-circle, triangle-sour, and green-bitter). Thus, while some crossmodal correspondences emerge during childhood, others may require additional time to develop, thereby highlighting the importance of understanding the cognitive mechanisms underlying crossmodal correspondences from an ontogenic perspective.

Synergistic Combination of Visual Features in Vision-Taste Crossmodal Correspondences.

There has been a rapid recent growth in academic attempts to summarise, understand, and predict the taste profile matching complex images that incorporate multiple visual design features. While there is now ample research to document the patterns of vision-taste correspondences involving individual visual features (such as colour and shape curvilinearity in isolation), little is known about the taste associations that may be primed when multiple visual features are presented simultaneously. This narrative historical review therefore presents an overview of the research that has examined, or provided insights into, the interaction of graphic elements in taste correspondences involving colour, shape attributes, texture, and other visual features. The empirical evidence is largely in line with the predictions derived from the proposed theories concerning the origins of crossmodal correspondences; the component features of a visual stimulus are observed to contribute substantially to its taste expectations. However, the taste associated with a visual stimulus may sometimes deviate from the taste correspondences primed by its constituent parts. This may occur when a new semantic meaning emerges as multiple features are displayed together. Some visual features may even provide contextual cues for observers, thus altering the gustatory information that they associate with an image. A theoretical framework is constructed to help more intuitively predict and conceptualise the overall influence on taste correspondences when visual features are processed together as a combined image.

Neural basis of sound-symbolic pseudoword-shape correspondences

Non-arbitrary mapping between the sound of a word and its meaning, termed sound symbolism, is commonly studied through crossmodal correspondences between sounds and visual shapes, e.g., auditory pseudowords, like ‘mohloh’ and ‘kehteh’, are matched to rounded and pointed visual shapes, respectively. Here, we used functional magnetic resonance imaging (fMRI) during a crossmodal matching task to investigate the hypotheses that sound symbolism (1) involves language processing; (2) depends on multisensory integration; (3) reflects embodiment of speech in hand movements. These hypotheses lead to corresponding neuroanatomical predictions of crossmodal congruency effects in (1) the language network; (2) areas mediating multisensory processing, including visual and auditory cortex; (3) regions responsible for sensorimotor control of the hand and mouth. Right-handed participants (n = 22) encountered audiovisual stimuli comprising a simultaneously presented visual shape (rounded or pointed) and an auditory pseudoword (‘mohloh’ or ‘kehteh’) and indicated via a right-hand keypress whether the stimuli matched or not. Reaction times were faster for congruent than incongruent stimuli. Univariate analysis showed that activity was greater for the congruent compared to the incongruent condition in the left primary and association auditory cortex, and left anterior fusiform/parahippocampal gyri. Multivoxel pattern analysis revealed higher classification accuracy for the audiovisual stimuli when congruent than when incongruent, in the pars opercularis of the left inferior frontal (Broca's area), the left supramarginal, and the right mid-occipital gyri. These findings, considered in relation to the neuroanatomical predictions, support the first two hypotheses and suggest that sound symbolism involves both language processing and multisensory integration.

Dynamic audio-visual correspondence in musicians and non-musicians

Most previous audio-visual crossmodal correspondence studies focused on static visual or auditory stimuli. Moreover, some researchers have found that music training can effectively improve the interaction between auditory and visual channels by enhancing neural plasticity. This study focused on whether crossmodal correspondence occurs when people face dynamic visual and auditory stimuli, and whether it is affected by musical training. Participants were asked to judge different changes in pitch (rise and drop) by showing them different patterns of visual circle motion (enlarged, reduced, and unchanged). The results revealed that the audio-visual congruent combinations (the pitch rise when the circle was enlarged and the pitch drop when the circle was reduced) significantly shortened participants’ response times, whereas the audio-visual neutral combinations (the unchanged shape) had the highest accuracy. Participants with musical training were faster than participants without musical training to judge pitch changes in conditions where the association between pitch and shape size was incongruent. These findings provide empirical evidence for dynamic audio-visual crossmodal correspondence and shed light on the bright prospect of using congruent audio-visual stimuli in animation.

Bidirectionality in multisensory perception: Examining the mutual influences between audition and taste

Previous research suggests that people reliably associate sounds and tastes. One important and often-cited implication of these crossmodal correspondences is the modulatory effect of sound on taste perception. In contrast, the potential impact of gustatory sensations on auditory perception has received less empirical attention. This paper presents the results of three experiments examining how listening to a sweet (vs. bitter) soundtrack shapes the sensory perception of bittersweet chocolate (Experiments 1a and 1b) and how the sweet (vs. bitter) taste of chocolate affects the perception of congruent sensory and hedonic attributes of a “bittersweet” soundtrack (Experiment 2). Experiment 1a manipulated the soundtrack type between participants and found no significant effect of music on taste perception. Experiment 1b followed a similar procedure but with a within-participants design. Here, the chocolate sample was perceived as sweeter and more positive when paired with the sweet soundtrack. In Experiment 2, tasting sweet chocolate shifted the evaluation of the bittersweet soundtrack toward higher sweetness and pleasantness and lower bitterness ratings. These findings suggest that sound-taste correspondences may have bidirectional effects on gustatory and auditory stimuli perception. However, the effects of audition on taste may depend on the direct contrast between soundtracks with different crossmodal profiles. These findings contribute to a better understanding of multisensory interactions between audition and taste. The implications for future research and the challenges to real-world interventions are discussed.

Using crossmodal correspondences as a tool in wine communication.

This research investigates consumer acceptance of alternative methods for communicating information about wine, focusing on the alignment between sensory attributes and consumer expectations. A survey was administered to wine enthusiasts to assess their attitudes toward crossmodal communication. The findings reveal significant associations between consumer behaviors and acceptance of alternative communication methods, highlighting the emerging field of crossmodal correspondences. These results suggest that leveraging crossmodal correspondences can enhance the match between a product's sensory qualities and consumer expectations, potentially reducing wine wastage resulting from unmet consumer preferences. These findings have implications for improving communication strategies in the wine industry and enhancing consumer experiences.

Congruency of color-sound crossmodal correspondence interacts with color and sound discrimination depending on color category.

People occasionally associate color (e.g., hue) with sound (e.g., pitch). Previous studies have reported color-sound associations, which are examples of crossmodal correspondences. However, the association between both semantic and perceptual factors with color/sound discrimination in crossmodal correspondence remains unclear. To clarify this, three psychological experiments were conducted, where Stroop tasks were used to assess automatic process on the association. We focused on the crossmodal correspondence between color (Experiment 1)/color word (Experiment 2) and sound. Participants discriminated the color/word or the sound presented simultaneously. The results showed the color-sound bidirectional enhancement/interference of the response by certain associations of the crossmodal correspondence (blue-drop and yellow-shiny) in both experiments. These results suggest that these Stroop effects were caused by the semantic factor (color category) and the perceptual factor (color appearance) was not necessary for the current results. In Experiment 3, response modulation by color labeling was investigated to clarify the influence of subjective labeling. Participants labeled a presented ambiguous color, which was a hue specification between two specific colors, by listening to the sound. The results revealed that the Stroop effect was caused only when the presented color was classified as the color related to the presented sound. This showed that subjective labeling played a role in the regulation of the effect of crossmodal correspondences. These findings should contribute to the explanation of crossmodal correspondences through semantic mediation.

Physicochemical features partially explain olfactory crossmodal correspondences

During the olfactory perception process, our olfactory receptors are thought to recognize specific chemical features. These features may contribute towards explaining our crossmodal perception. The physicochemical features of odors can be extracted using an array of gas sensors, also known as an electronic nose. The present study investigates the role that the physicochemical features of olfactory stimuli play in explaining the nature and origin of olfactory crossmodal correspondences, which is a consistently overlooked aspect of prior work. Here, we answer the question of whether the physicochemical features of odors contribute towards explaining olfactory crossmodal correspondences and by how much. We found a similarity of 49% between the perceptual and the physicochemical spaces of our odors. All of our explored crossmodal correspondences namely, the angularity of shapes, smoothness of textures, perceived pleasantness, pitch, and colors have significant predictors for various physicochemical features, including aspects of intensity and odor quality. While it is generally recognized that olfactory perception is strongly shaped by context, experience, and learning, our findings show that a link, albeit small (6–23%), exists between olfactory crossmodal correspondences and their underlying physicochemical features.

Exploring crossmodal correspondences for future research in human movement augmentation.

"Crossmodal correspondences" are the consistent mappings between perceptual dimensions or stimuli from different sensory domains, which have been widely observed in the general population and investigated by experimental psychologists in recent years. At the same time, the emerging field of human movement augmentation (i.e., the enhancement of an individual's motor abilities by means of artificial devices) has been struggling with the question of how to relay supplementary information concerning the state of the artificial device and its interaction with the environment to the user, which may help the latter to control the device more effectively. To date, this challenge has not been explicitly addressed by capitalizing on our emerging knowledge concerning crossmodal correspondences, despite these being tightly related to multisensory integration. In this perspective paper, we introduce some of the latest research findings on the crossmodal correspondences and their potential role in human augmentation. We then consider three ways in which the former might impact the latter, and the feasibility of this process. First, crossmodal correspondences, given the documented effect on attentional processing, might facilitate the integration of device status information (e.g., concerning position) coming from different sensory modalities (e.g., haptic and visual), thus increasing their usefulness for motor control and embodiment. Second, by capitalizing on their widespread and seemingly spontaneous nature, crossmodal correspondences might be exploited to reduce the cognitive burden caused by additional sensory inputs and the time required for the human brain to adapt the representation of the body to the presence of the artificial device. Third, to accomplish the first two points, the benefits of crossmodal correspondences should be maintained even after sensory substitution, a strategy commonly used when implementing supplementary feedback.

The influence of Bouba- and Kiki-like shape on perceived taste of chocolate pieces.

In this paper, we present the findings of a study investigating the impact of shape on the taste perception of chocolate. Previous research has explored the influence of various sensory information on taste perception, but there has been little focus on the effect of food shape being eaten on taste perception. To explore this, we focused on the Bouba-Kiki effect, illustrating an interaction between shape and several modalities, and investigated the effect of Bouba- and Kiki-shaped (rounded and angular) foods eaten on taste perception. We utilized a 3D food printer to produce four different shapes of chocolate pieces based on the Bouba-Kiki. Participants tasted each piece and completed a chocolate flavor questionnaire. With Bayesian analysis, we determined that the Bouba-shaped chocolate pieces were perceived as sweeter than the Kiki-shaped ones, supporting earlier studies on crossmodal correspondences between shape and taste perception. However, there were no significant differences in ratings of other tastes, such as sourness and bitterness. Our research indicates that shape can affect taste perception during consumption and suggests that 3D food printers offer an opportunity to design specific shapes that influence taste experiences.