Cross-modal Associations Research Articles

Timing is everything: Onset timing moderates the crossmodal influence of background sound on taste perception.

Recent evidence demonstrates that the presentation of crossmodally corresponding auditory stimuli can modulate the taste and hedonic evaluation of various foods (an effect often called "sonic seasoning"). To further understand the mechanism underpinning such crossmodal effects, the time at which a soundtrack was presented relative to tasting was manipulated in a series of experiments. Participants heard two soundtracks corresponding to sweet and bitter tastes either exclusively during or after chocolate tasting (Experiment 1) or during and before chocolate tasting (Experiment 2). The results revealed that the soundtracks affected chocolate taste ratings only if they were presented before or during tasting but not if they were heard after tasting. Moreover, participants' individual soundtrack-taste association mediated the strength of the sonic seasoning effect. These results therefore imply that the modulatory effect of sound on taste was not driven by retrospective interpretation of the taste experience, but by mechanisms such as priming and crossmodal association. Taken together, these studies demonstrate the complex interplay of cognitive mechanisms that likely underlie sonic seasoning effects. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Do visual and auditory stimulus-specific response modulation reflect different mechanisms of neocortical plasticity?

Stimulus response modulation (SRM) of sensory evoked potentials represents a promising method as a non-invasive index of long-term potentiation (LTP)-like synaptic plasticity in the human sensory cortices. As of today, however, no consensus exists regarding which experimental parameters elicit the most robust SRM response. The aim of the current study was twofold; firstly, we aimed to replicate former studies demonstrating visual SRM in healthy adults. Second, we integrated visual and auditory stimuli within the same SRM recording session to assay potential cross-modal associations. Such an association between modalities would strengthen the assumption that the SRM effect reflects common mechanisms underlying synaptic plasticity rather than reflecting modality-specific phenomena. A replication of previous findings showing robust potentiation of the visual evoked potential was evident, supporting the majority of previous work using similar paradigms, lending further support to the notion that high-frequent visual stimulation is a viable probe into LTP-like synaptic plasticity in the human visual cortex. The auditory evoked potentials (AEPs) did not, however, fully replicate previous work, though a significant increase of temporally later AEP components was found. In contrast to our hypothesis, there were no significant within-subject cross-modality correlations between the visual and auditory SRM. This lack of significant association might suggest that auditory and visual SRM depend on different mechanisms, and that further SRM studies on non-invasive LTP-like synaptic plasticity should focus on optimizing paradigms within the visual modality.

Assessing the Role of Emotional Mediation in Explaining Crossmodal Correspondences Involving Musical Stimuli.

A wide variety of crossmodal correspondences, defined as the often surprising connections that people appear to experience between simple features, attributes, or dimensions of experience, either physically present or else merely imagined, in different sensory modalities, have been demonstrated in recent years. However, a number of crossmodal correspondences have also been documented between more complex (i.e., multi-component) stimuli, such as, for example, pieces of music and paintings. In this review, the extensive evidence supporting the emotional mediation account of the crossmodal correspondences between musical stimuli (mostly pre-recorded short classical music excerpts) and visual stimuli, including colour patches through to, on occasion, paintings, is critically evaluated. According to the emotional mediation account, it is the emotional associations that people have with stimuli that constitutes one of the fundamental bases on which crossmodal associations are established. Taken together, the literature that has been published to date supports emotional mediation as one of the key factors underlying the crossmodal correspondences involving emotionally-valenced stimuli, both simple and complex.

Crossmodal Associations between Cinema with Elements of Comic and Tragic and Texture Touch

Crossmodal associations are systematic links between a stimulus from one modality (e.g., color) and a stimulus from another modality (e.g., sound). The present study investigated crossmodal associations between cinema (a complex visual and auditory stimulus) and texture touch. Participants watched fragments of movies with comic and tragic elements. They also touched different textures (e.g., silk, fur, marble) and chose the textures that were most/least consistent with the films. We found systematic links between the films and the textures associated with them. Films with elements of tragedy were associated with granite, marble and glass, while films with elements of comedy were associated with glass pebbles, plasticine and a slime toy. Similar emotional and semantic evaluations of textures and films can partly explain these crossmodal associations. Significant correlations were found between the direct emotional evaluation of films and indirect evaluation through the cinema-haptic perception index for the following scales: dirty, disgusting, pleasant and happy.

Concurrent EEG- and fMRI-derived functional connectomes exhibit linked dynamics

Long-range connectivity has become the most studied feature of human functional Magnetic Resonance Imaging (fMRI), yet the spatial and temporal relationship between its whole-brain dynamics and electrophysiological connectivity remains largely unknown. FMRI-derived functional connectivity exhibits spatial reconfigurations or time-varying dynamics at infraslow (<0.1Hz) speeds. Conversely, electrophysiological connectivity is based on cross-region coupling of fast oscillations (~1–100Hz). It is unclear whether such fast oscillation-based coupling varies at infraslow speeds, temporally coinciding with infraslow dynamics across the fMRI-based connectome. If so, does the association of fMRI-derived and electrophysiological dynamics spatially vary over the connectome across the functionally distinct electrophysiological oscillation bands? In two concurrent electroencephalography (EEG)-fMRI resting-state datasets, oscillation-based coherence in all canonical bands (delta through gamma) indeed reconfigured at infraslow speeds in tandem with fMRI-derived connectivity changes in corresponding region-pairs. Interestingly, irrespective of EEG frequency-band the cross-modal tie of connectivity dynamics comprised a large proportion of connections distributed across the entire connectome. However, there were frequency-specific differences in the relative strength of the cross-modal association. This association was strongest in visual to somatomotor connections for slower EEG-bands, and in connections involving the Default Mode Network for faster EEG-bands. Methodologically, the findings imply that neural connectivity dynamics can be reliably measured by fMRI despite heavy susceptibility to noise, and by EEG despite shortcomings of source reconstruction. Biologically, the findings provide evidence that contrast with known territories of oscillation power, oscillation coupling in all bands slowly reconfigures in a highly distributed manner across the whole-brain connectome.

DeepDance: Music-to-Dance Motion Choreography With Adversarial Learning

The creation of improvised dancing choreographies is an important research field of cross-modal analysis. A key point of this task is how to effectively create and correlate music and dance with a probabilistic one-to-many mapping, which is essential to create realistic dances of various genres. To address this issue, we propose a GAN-based cross-modal association framework, DeepDance, which correlates two different modalities (dance motion and music) together, aiming at creating the desired dance sequence in terms of the input music. Its generator is to predictively produce the dance movements best-fit to current music piece by learning from examples. In another hand, its discriminator acts as an external evaluation from the audience and judges the whole performance. The generated dance movements and the corresponding input music are considered to be well-matched if the discriminator cannot distinguish the generated movements from the training samples according to the estimated probability. By adding motion consistency constraints in our loss function, the proposed framework is able to create long realistic dance sequences. To alleviate the problem of expensive and inefficient data collection, we propose an effective approach to create a large-scale dataset, YouTube-Dance3D, from open data source. Extensive experiments on currently available music-dance datasets and our YouTube-Dance3D dataset demonstrate that our approach effectively captures the correlation between music and dance and can be used to choreograph appropriate dance sequences.

“Bitter Touch”: Cross-modal associations between hand-feel touch and gustatory cues in the context of coffee consumption experience

Since a variety of packaging and containers have become commercially available in the market, there has been a rapidly growing interest in the influences of hand-feel touch cues on consumer perceptions and emotional responses toward food and beverage products. This study aimed to determine whether hand-feel touch cues of cup-sleeve materials could be associated with imagined (Study 1) and consumed (Study 2) basic tastes, and thereby affect the perception of brewed coffee (Study 3). Participants were asked to evaluate twelve different cup-sleeve materials with respect to evoked emotions and their degree of matching with each of the four basic taste qualities and brewed coffee-related flavor attributes (Study 1). Individual cup-sleeve materials were found to be more associated with specific taste qualities, coffee-related flavors, and emotions. Hand-feel touch cues of different sleeves were also found to be associated with taste qualities consumed (Study 2). For example, towel, linen, stainless steel, and cardboard materials were matched with sweet, salty, sour, and bitter taste qualities, respectively. Specific physical characteristics of cup-sleeve materials were found to be involved in mediating such cross-modal associations between hand-feel touch and taste cues. In Study 3, participants were asked to evaluate brewed coffee samples in paper cups both with cardboard sleeves and those made from the other test materials (towel, linen, and stainless steel). While participants rated black coffee samples with a towel sleeve less bitter than those with a cardboard sleeve, such differences were not observed in other pairwise comparisons. In conclusion, this study provided empirical evidence that hand-feel touch cues can be associated with specific taste or coffee-related flavor attributes, thereby modulating consumer perception of brewed coffee.

The Color Nutrition Information Paradox: Effects of Suggested Sugar Content on Food Cue Reactivity in Healthy Young Women

Color nutrition information (CNI) based on a traffic light system conveys information about food quality with a glance. The color red typically indicates detrimental food characteristics (e.g., very high sugar content) and aims at inhibiting food shopping and consumption. Red may, however, also elicit cross-modal associations with sweet taste, which is a preferable food characteristic. We conducted two experiments. An eye-tracking study investigated whether CNI has an effect on cue reactivity (dwell time, saccadic latency, wanting/liking) for sweet foods. The participants were presented with images depicting sweets (e.g., cake). Each image was preceded by a colored circle that informed about the sugar content of the food (red = high, green = low, gray = unknown). It was tested whether the red circle would help the participants to direct their gaze away from the ‘high sugar’ item. A second experiment investigated whether colored prime circles (red, green, gray) without nutrition information would influence the assumed sweetness of a food. In Experiment 1, CNI had the opposite of the intended effect. Dwell time and saccadic latency were higher for food items preceded by a red compared to a green circle. This unintended response was positively associated with participants’ liking of sweet foods. CNI did not change the wanting/liking of the displayed foods. In Experiment 2, we found no evidence for color priming on the assumed sweetness of food. Our results question whether CNI is helpful to influence initial cue reactivity toward sweet foods.

Investigation of Main Effect of Scent in Cross-modal Association between Music and Scent

Investigation of Main Effect of Scent in Cross-modal Association between Music and Scent

Nasal Sounds are Lighter and More Yellowish than Glottal Sounds: Cross-modal Associations between Consonant Sounds and Colors

Nasal Sounds are Lighter and More Yellowish than Glottal Sounds: Cross-modal Associations between Consonant Sounds and Colors

Multi-Modal Memory Enhancement Attention Network for Image-Text Matching

Image-text matching is an attractive research topic in the community of vision and language. The key element to narrow the “heterogeneity gap” between visual and textual data lies in how to learn powerful and robust representations for both modalities. This paper proposes to alleviate this issue to achieve the fine-grained visual-textual alignment from two aspects: exploiting attention mechanism to locate the semantically meaningful portion and leveraging the memory network to capture the long-term contextual knowledge. Unlike most existing studies sorely focus on exploring the cross-modal associations at the fragment level, our designed Collaborative Dual Attention (CDA) module is able to model the semantic interdependencies from both perspectives of fragment and channel. Furthermore, considering the usage of long-term contextual knowledge contributes to compensate for detailed semantics concealed in the rarely appeared image-text pairs, we present to learn the joint representations by constructing a Multi-Modal Memory Enhancement (M3E) module. Specifically, it sequentially restores the intra-modal and multi-modal information into the memory items, and they conversely persistently memorize cross-modal shared semantics to improve the latent embeddings. By incorporating both CDA and M3E modules into a deep architecture, our approach generates more semantically consistent embeddings for representing images and texts. Extensive experiments demonstrate our model can achieve the state-of-the-art results on two public benchmark datasets.

Gender (Conformity) Matters: Cross-Dimensional and Cross-Modal Associations in Sexual Orientation Perception

Although sexual orientation (SO) is perceptually ambiguous, people are able to detect it with above-chance accuracy from faces and, sometimes, from voices. Despite a multitude of “gaydar” studies, it is unclear (1) whether vocal or facial signals carry more SO information, (2) whether raters refer to target’s SO instead of gender-role conformity when forming SO impressions, and (3) whether there are any differences for female and male targets. We collected face photographs, voice recordings, and self-reported gender-role conformity of 18 lesbian/gay and straight female and male target persons each. Study 1 (rating of SO) showed that faces led to higher accuracies than voices, which was especially true for female targets. Study 2 (rating of gender-role conformity) showed that the link between self-reported and attributed SO was mediated by self-reported and attributed gender-role conformity. Results support the centrality of gender-role conformity, more than that of SO, in impression formation.

Color sound symbolism in natural languages

abstractThis paper investigates the underlying cognitive processes of sound–color associations by connecting perceptual evidence from research on cross-modal correspondences to sound symbolic patterns in the words for colors in natural languages. Building upon earlier perceptual experiments, we hypothesized that sonorous and bright phonemes would be over-represented in the words for bright and saturated colors. This hypothesis was tested on eleven color words and related concepts (red–green, yellow–blue, black–white, gray, night–day, dark–light) from 245 language families. Textual data was transcribed into the International Phonetic Alphabet (IPA), and each phoneme was described acoustically using high-quality IPA recordings. These acoustic measurements were then correlated with the luminance and saturation of each color obtained from cross-linguistic color-naming data in the World Color Survey. As expected, vowels with high brightness and sonority ratings were over-represented in the words for colors with high luminance, while sonorous consonants were more common in the words for saturated colors. We discuss these results in relation to lexicalization patterns and the links between iconicity and perceptual cross-modal associations.

Chaos based non-linear cognitive study of different stimulus in the cross-modal perspective

The relationship between color and music as part of the complex system consisting of visual and auditory domain has been investigated in this study. As both the stimulus forms are processed in the same part of the human body, i.e., the brain, it will be really interesting to examine whether they share a similarity in perception. Needless to say that color and music both have strong impact on emotion and feelings & also a few studies have been reported in literature to explore causal relationship between color and emotion. This work reports a neuro-cognitive study on response of brain to two different stimulus and their cross-modal associations. In this study the correlation between emotional arousal and the effect of audio and visual stimuli has been studied from a new perspective. 93 participants were asked to hear 6 different music pieces (each of 30 s duration). The type of emotion elicited by different music pieces were identified by the participants from a given collection of possible emotional responses. Then they are asked to assign a color associating the emotion from a given color wheel (structured according to Munsell color system/RGB color space). Each color, associated with a particular music piece, is a mixture of specific Red, Green and Blue values (RGB triplet) and has a specific HEX number (hexadecimal representation), which is recorded for each response. Then, the musical pieces used were further zoomed with the help of fractal technique to identify different emotions related to music in a quantitative approach. Here, to analyze the complexity of the sound signal (which are non-stationary and scale varying in nature), we have used Multifractal detrended fluctuation analysis (MFDFA), which is capable of determining multifractal scaling behavior of non-stationary time series. From the experimental data, it is seen that the visual and emotional response to the auditory stimulus follows a specific trend which is directly related to the stimulus complexity.

Coarse to Fine Audio-Visual Size Correspondences Develop During Primary School Age.

Developmental studies have shown that children can associate visual size with non-visual and apparently unrelated stimuli, such as pure tone frequencies. Most research to date has focused on audio-visual size associations by showing that children can associate low pure tone frequencies with large objects, and high pure tone frequencies with small objects. Researchers relate these findings to coarser association, i.e., less precise associations for which binary categories of stimuli are used such as in the case of low versus high frequencies and large versus small visual stimuli. This study investigates how finer, more precise, crossmodal audio-visual associations develop during primary school age (from 6 to 11 years old). To unveil such patterns, we took advantage of a range of auditory pure tones and tested how primary school children match sounds with visually presented shapes. We tested 66 children (6–11 years old) in an audio-visual matching task involving a range of pure tone frequencies. Visual stimuli were circles or angles of different sizes. We asked participants to indicate the shape matching the sound. All children associated large objects/angles with low pitch, and small objects/angles with high pitch sounds. Interestingly, older children made greater use of intermediate visual sizes to provide their responses. Indeed, audio-visual associations for finer differences between stimulus features such as size and pure tone frequencies, may develop later depending on the maturation of supramodal size perception processes. Considering our results, we suggest that audio-visual size correspondences can be used for educational purposes by aiding the discrimination of sizes, including angles of different aperture. Moreover, their use should be shaped according to children’s specific developmental stage.

A red code triggers an unintended approach motivation toward sweet ultra-processed foods: Possible implications for front-of-pack labels

Front-of-package labels (FOPL) are recommended to reduce consumer intake of ultra-processed food products (UPP). The multiple traffic-light label is one example of FOPL that indicates the content of target nutrients in products by displaying red (high), amber (medium), and/or green (low) color-coding. The red code may implicitly enhance sweetness perception and approach dispositions toward sweet UPP via cross-modal visual-taste interactions. We conducted two experiments to examine the possibility of contradictory influence of explicitly learned and implicit cross-modal associations on the emotional responses evoked by UPP pictures. In both experiments, we first explicitly associated the color codes with health-related meanings. In Experiment I (n = 78), a psychometric tool estimated the emotional responses (pleasantness and arousal ratings) evoked by UPP pictures when preceded by red, amber, or green color-codes. In Experiment II (n = 24), we recorded participants’ electrocortical brain activity to assess the early posterior negativity (EPN) component as an index of the emotional responses to UPP. The reported pleasantness (Experiment I) and the EPN amplitude (Experiment II) were greater for sweet UPP relative to salty UPP when primed with red codes but not when primed with green or amber. A red code increased positive emotions toward sweet UPP despite its explicit association with increased health-risks. Thus, the use of multiple traffic-lights might lead to an unintended implicit approach behavior toward sweet UPP. Designers, researchers, and policy makers may consider color-taste cross-modal associations when designing, testing, and applying FOPL.

Perception it is: Processing level in multisensory selection

When repeatedly exposed to simultaneously presented stimuli, associations between these stimuli are nearly always established, both within as well as between sensory modalities. Such associations guide our subsequent actions and may also play a role in multisensory selection. Thus, crossmodal associations (i.e., associations between stimuli from different modalities) learned in a multisensory interference task might affect subsequent information processing. The aim of this study was to investigate the processing level of multisensory stimuli in multisensory selection by means of crossmodal aftereffects. Either feature or response associations were induced in a multisensory flanker task while the amount of interference in a subsequent crossmodal flanker task was measured. The results of Experiment 1 revealed the existence of crossmodal interference after multisensory selection. Experiments 2 and 3 then went on to demonstrate the dependence of this effect on the perceptual associations between features themselves, rather than on the associations between feature and response. Establishing response associations did not lead to a subsequent crossmodal interference effect (Experiment 2), while stimulus feature associations without response associations (obtained by changing the response effectors) did (Experiment 3). Taken together, this pattern of results suggests that associations in multisensory selection, and the interference of (crossmodal) distractors, predominantly work at the perceptual, rather than at the response, level.

MixMash

This article presents MixMash, an interactive tool which streamlines the process of music mashup creation by assisting users in the process of finding compatible music from a large collection of audio tracks. It extends the harmonic mixing method by Bernardes, Davies and Guedes with novel degrees of harmonic, rhythmic, spectral, and timbral similarity metrics. Furthermore, it revises and improves some interface design limitations identified in the former model software implementation. A new user interface design based on cross-modal associations between musical content analysis and information visualisation is presented. In this graphic model, all tracks are represented as nodes where distances and edge connections display their harmonic compatibility as a result of a force-directed graph. Besides, a visual language is defined to enhance the tool's usability and foster creative endeavour in the search of meaningful music mashups.

Do sound symbolism effects for written words relate to individual phonemes or to phoneme features?

abstractThe sound of words has been shown to relate to the meaning that the words denote, an effect that extends beyond morphological properties of the word. Studies of these sound-symbolic relations have described this iconicity in terms of individual phonemes, or alternatively due to acoustic properties (expressed in phonological features) relating to meaning. In this study, we investigated whether individual phonemes or phoneme features best accounted for iconicity effects. We tested 92 participants’ judgements about the appropriateness of 320 nonwords presented in written form, relating to 8 different semantic attributes. For all 8 attributes, individual phonemes fitted participants’ responses better than general phoneme features. These results challenge claims that sound-symbolic effects for visually presented words can access broad, cross-modal associations between sound and meaning, instead the results indicate the operation of individual phoneme to meaning relations. Whether similar effects are found for nonwords presented auditorially remains an open question.

Verbal and Cross-Modal Ratings of Music: Validation and Application of an Icon-Based Rating Scale.

Can music be rated consistently using nonverbal descriptors such as colours and temperatures? 144 participants rated 6 experimenter-selected and 2 self-selected pieces of music along 15 bipolar icon (graphic) scales intended to portray emotions, and sensory experiences consisting of colour, temperature, shape, speed, texture, and weight. Participants also rated the same pieces using bipolar verbal scales which aimed to encompass the concepts represented by the icons (e.g., the word “red” for the colour red). Furthermore, the icons themselves were subjected to open-ended verbal labelling to validate the icon scale. Colour icons spontaneously evoked a cross-modal association on 67% of occasions: blue being cool, and red/orange being warm or hot, and the icon scale had overall good face validity. Music regularly and consistently evoked multisensory associations (using the icon scale) including shapes, colours, weight, and temperatures, in addition to emotions. Cross-modal perception is indicative of music’s character rather than the enjoyment of the music. The icon scale provides new insights into music perception and for applications where language skill may limit participant expression.