Cross-modal Associations Research Articles

Cross-modal Association between Auditory and Visuospatial Information in Mandarin Tone Perception in Noise by Native and Non-native Perceivers.

Speech perception involves multiple input modalities. Research has indicated that perceivers establish cross-modal associations between auditory and visuospatial events to aid perception. Such intermodal relations can be particularly beneficial for speech development and learning, where infants and non-native perceivers need additional resources to acquire and process new sounds. This study examines how facial articulatory cues and co-speech hand gestures mimicking pitch contours in space affect non-native Mandarin tone perception. Native English as well as Mandarin perceivers identified tones embedded in noise with either congruent or incongruent Auditory-Facial (AF) and Auditory-FacialGestural (AFG) inputs. Native Mandarin results showed the expected ceiling-level performance in the congruent AF and AFG conditions. In the incongruent conditions, while AF identification was primarily auditory-based, AFG identification was partially based on gestures, demonstrating the use of gestures as valid cues in tone identification. The English perceivers’ performance was poor in the congruent AF condition, but improved significantly in AFG. While the incongruent AF identification showed some reliance on facial information, incongruent AFG identification relied more on gestural than auditory-facial information. These results indicate positive effects of facial and especially gestural input on non-native tone perception, suggesting that cross-modal (visuospatial) resources can be recruited to aid auditory perception when phonetic demands are high. The current findings may inform patterns of tone acquisition and development, suggesting how multi-modal speech enhancement principles may be applied to facilitate speech learning.

Environmental sound priming: Does negation modify N400 cross-modal priming effects?

Human information processing is incredibly fast and flexible. In order to survive, the human brain has to integrate information from various sources and to derive a coherent interpretation, ideally leading to adequate behavior. In experimental setups, such integration phenomena are often investigated in terms of cross-modal association effects. Interestingly, to date, most of these cross-modal association effects using linguistic stimuli have shown that single words can influence the processing of non-linguistic stimuli, and vice versa. In the present study, we were particularly interested in how far linguistic input beyond single words influences the processing of non-linguistic stimuli; in our case, environmental sounds. Participants read sentences either in an affirmative or negated version: for example: "The dog does (not) bark". Subsequently, participants listened to a sound either matching or mismatching the affirmative version of the sentence ('woof' vs. 'meow', respectively). In line with previous studies, we found a clear N400-like effect during sound perception following affirmative sentences. Interestingly, this effect was identically present following negated sentences, and the negation operator did not modulate the cross-modal association effect observed between the content words of the sentence and the sound. In summary, these results suggest that negation is not incorporated during information processing in a manner that word-sound association effects would be influenced.

Musical pitch classes have rainbow hues in pitch class-color synesthesia

Synesthesia, an anomalous blending of senses in which stimulation of one sensory modality produces sensation in a different modality, provides a unique opportunity to study how multimodal information is represented in the human brain. We investigated how pitch classes (do, re, mi, etc.) are associated with the three dimensions of color (hue, saturation, and value/brightness) in 15 subjects who possessed “pitch class-color synesthesia”. Across-subject averaging of reported colors revealed that pitch classes have rainbow hues, beginning with do-red, re-yellow, and so forth, ending with si-violet, accompanied by a decrease in saturation. Enharmonic pitch classes that referred to the same pitch class with a different name produced color sensations according to the name of the base pitch class, e.g., a reddish color for do-sharp and a yellowish color for re-flat. Thus the main factor producing color sensations was the name, not the sound, of the note; behavioral experiments corroborated this interpretation. Pitch class-color synesthesia represents a newly described type of synesthesia that is distinct from the well-known crossmodal association between pitch height and value/brightness. Findings suggest that the two dimensions of musical pitch, pitch class and pitch height, are mapped to the hue-saturation plane and the value/brightness dimension of color, respectively.

Sound iconicity of abstract concepts: Place of articulation is implicitly associated with abstract concepts of size and social dominance.

The concept of sound iconicity implies that phonemes are intrinsically associated with non-acoustic phenomena, such as emotional expression, object size or shape, or other perceptual features. In this respect, sound iconicity is related to other forms of cross-modal associations in which stimuli from different sensory modalities are associated with each other due to the implicitly perceived correspondence of their primal features. One prominent example is the association between vowels, categorized according to their place of articulation, and size, with back vowels being associated with bigness and front vowels with smallness. However, to date the relative influence of perceptual and conceptual cognitive processing on this association is not clear. To bridge this gap, three experiments were conducted in which associations between nonsense words and pictures of animals or emotional body postures were tested. In these experiments participants had to infer the relation between visual stimuli and the notion of size from the content of the pictures, while directly perceivable features did not support–or even contradicted–the predicted association. Results show that implicit associations between articulatory-acoustic characteristics of phonemes and pictures are mainly influenced by semantic features, i.e., the content of a picture, whereas the influence of perceivable features, i.e., size or shape, is overridden. This suggests that abstract semantic concepts can function as an interface between different sensory modalities, facilitating cross-modal associations.

Seeing odors in color: Cross-modal associations in children and adults from two cultural environments

We investigated the occurrence and underlying processes of odor–color associations in French and American 6- to 10-year-old children (n = 386) and adults (n = 137). Nine odorants were chosen according to their familiarity to either cultural group. Participants matched each odor with a color, gave hedonic and familiarity judgments, and identified each odor. By 6 years of age, children displayed culture-specific odor–color associations, but age differences were noted in the type of associations. Children and adults in both cultural groups shared common associations and formed associations that were unique to their environment, underscoring the importance of exposure learning in odor–color associations.

How >:( is Bizet? Icon ratings of music

Can music be rated consistently using icon descriptors without verbal mediation? Sixty-eight participants rated six experimenter- and two self-selected pieces of music along 15 bipolar icon scales intended to portray emotions, and sensory experiences involving color, temperature, shape, speed, texture, and weight. Several replications were reported, including Mozart being blue, Brahms being soft and Bizet being takete (a jagged shape). Crossmodal associations with individual pieces were similar to those reported in previous studies, but self-selected liked and disliked musics did not evoke as many such associations, leading to the conclusion that crossmodal perception may be indicative of music character more so than it is of hedonic tone. The similarity of results in the present study and previous research suggests that icon scales may provide a convenient alternative to sensory scales in various modalities that are difficult to reproduce via current conventional computer interface technology.

Serotonin gating of cortical and thalamic glutamate inputs onto principal neurons of the basolateral amygdala

The basolateral amygdala (BLA) is a key site for crossmodal association of sensory stimuli and an important relay in the neural circuitry of emotion. Indeed, the BLA receives substantial glutamatergic inputs from multiple brain regions including the prefrontal cortex and thalamic nuclei. Modulation of glutamatergic transmission in the BLA regulates stress- and anxiety-related behaviors. Serotonin (5-HT) also plays an important role in regulating stress-related behavior through activation of both pre- and postsynaptic 5-HT receptors. Multiple 5-HT receptors are expressed in the BLA, where 5-HT has been reported to modulate glutamatergic transmission. However, the 5-HT receptor subtype mediating this effect is not yet clear. The aim of this study was to use patch-clamp recordings from BLA neurons in an ex vivo slice preparation to examine 1) the effect of 5-HT on extrinsic sensory inputs, and 2) to determine if any pathway specificity exists in 5-HT regulation of glutamatergic transmission. Two independent input pathways into the BLA were stimulated: the external capsule to mimic cortical input, and the internal capsule to mimic thalamic input. Bath application of 5-HT reversibly reduced the amplitude of evoked excitatory postsynaptic currents (eEPSCs) induced by stimulation of both pathways. The decrease was associated with an increase in the paired-pulse ratio and coefficient of variation of eEPSC amplitude, suggesting 5-HT acts presynaptically. Moreover, the effect of 5-HT in both pathways was mimicked by the selective 5-HT1B receptor agonist CP93129, but not by the 5-HT1A receptor agonist 8-OH DPAT. Similarly the effect of exogenous 5-HT was blocked by the 5-HT1B receptor antagonist GR55562, but not affected by the 5-HT1A receptor antagonist WAY 100635 or the 5-HT2 receptor antagonists pirenperone and MDL 100907. Together these data suggest 5-HT gates cortical and thalamic glutamatergic inputs into the BLA by activating presynaptic 5-HT1B receptors.

Neural correlates of visuo-tactile crossmodal paired-associate learning and memory in humans

Studies have indicated that a cortical sensory system is capable of processing information from different sensory modalities. However, it still remains unclear when and how a cortical system integrates and retains information across sensory modalities during learning. Here we investigated the neural dynamics underlying crossmodal associations and memory by recording event-related potentials (ERPs) when human participants performed visuo-tactile (crossmodal) and visuo-visual (unimodal) paired-associate (PA) learning tasks. In a trial of the tasks, the participants were required to explore and learn the relationship (paired or non-paired) between two successive stimuli. EEG recordings revealed dynamic ERP changes during participants’ learning of paired-associations. Specifically, (1) the frontal N400 component showed learning-related changes in both unimodal and crossmodal tasks but did not show any significant difference between these two tasks, while the central P400 displayed both learning changes and task differences; (2) a late posterior negative slow wave (LPN) showed the learning effect only in the crossmodal task; (3) alpha-band oscillations appeared to be involved in crossmodal working memory. Additional behavioral experiments suggested that these ERP components were not relevant to the participants’ familiarity with stimuli per se. Further, by shortening the delay length (from 1300ms to 400ms or 200 ms) between the first and second stimulus in the crossmodal task, declines in participants’ task performance were observed accordingly. Taken together, these results provide insights into the cortical plasticity (induced by PA learning) of neural networks involved in crossmodal associations in working memory.

Hot and Cold Smells: Odor-Temperature Associations across Cultures.

It is often assumed odors are associated with hot and cold temperature, since odor processing may trigger thermal sensations, such as coolness in the case of mint. It is unknown, however, whether people make consistent temperature associations for a variety of everyday odors, and, if so, what determines them. Previous work investigating the bases of cross-modal associations suggests a number of possibilities, including universal forces (e.g., perception), as well as culture-specific forces (e.g., language and cultural beliefs). In this study, we examined odor-temperature associations in three cultures—Maniq (N = 11), Thai (N = 24), and Dutch (N = 24)—who differ with respect to their cultural preoccupation with odors, their odor lexicons, and their beliefs about the relationship of odors (and odor objects) to temperature. Participants matched 15 odors to temperature by touching cups filled with hot or cold water, and described the odors in their native language. The results showed no consistent associations among the Maniq, and only a handful of consistent associations between odor and temperature among the Thai and Dutch. The consistent associations differed across the two groups, arguing against their universality. Further analysis revealed cross-modal associations could not be explained by language, but could be the result of cultural beliefs.

Dissociating the time courses of the cross-modal semantic priming effects elicited by naturalistic sounds and spoken words

The present study compared the time courses of the cross-modal semantic priming effects elicited by naturalistic sounds and spoken words on visual picture processing. Following an auditory prime, a picture (or blank frame) was briefly presented and then immediately masked. The participants had to judge whether or not a picture had been presented. Naturalistic sounds consistently elicited a cross-modal semantic priming effect on visual sensitivity (d') for pictures (higher d' in the congruent than in the incongruent condition) at the 350-ms rather than at the 1,000-ms stimulus onset asynchrony (SOA). Spoken words mainly elicited a cross-modal semantic priming effect at the 1,000-ms rather than at the 350-ms SOA, but this effect was modulated by the order of testing these two SOAs. It would therefore appear that visual picture processing can be rapidly primed by naturalistic sounds via cross-modal associations, and this effect is short lived. In contrast, spoken words prime visual picture processing over a wider range of prime-target intervals, though this effect was conditioned by the prior context.

Does Language Influence the Vertical Representation of Auditory Pitch and Loudness?

Higher frequency and louder sounds are associated with higher positions whereas lower frequency and quieter sounds are associated with lower locations. In English, “high” and “low” are used to label pitch, loudness, and spatial verticality. By contrast, different words are preferentially used, in Catalan and Spanish, for pitch (high: “agut/agudo”; low: “greu/grave”) and for loudness/verticality (high: “alt/alto”; low: “baix/bajo”). Thus, English and Catalan/Spanish differ in the spatial connotations for pitch. To analyze the influence of language on these crossmodal associations, a task was conducted in which English and Spanish/Catalan speakers had to judge whether a tone was higher or lower (in pitch or loudness) than a reference tone. The response buttons were located at crossmodally congruent or incongruent positions with respect to the probe tone. Crossmodal correspondences were evidenced in both language groups. However, English speakers showed greater effects for pitch, suggesting an influence of linguistic background.

Musical mode and visual-spatial cross-modal associations in infants and adults

The classification of major and minor musical stimuli along five dichotomous scales (happy–sad, pleasant–unpleasant, up–down, light–dark, and warm–cold colors) was investigated in two studies involving 51 children aged 4–6 years, and 168 adults. Musical stimuli were six chords and six harmonized melodies differing in mode (major, minor). Furthermore, fluid intelligence was assessed in both infants and adults. The associations between major mode and happiness and between minor mode and sadness increased from a proportion of 58% at the age of 4, to 61% at the age of 5, 72% at the age of 6, and 92% in adults. The major–up, minor–down associations were 62% in both the 5- and 6-year-olds, and increased to 84% in adults, while the major–light, minor–dark associations increased from 59% at the age of 5 to 72% in 6-year-olds and 92% in adults. Warm–cold colors were systematically associated with, respectively, major and minor stimuli in adults but not in children. Fluid intelligence was strongly related to the child’s ability to associate happy and sad faces to major and minor musical stimuli.

When music is salty: The crossmodal associations between sound and taste.

Here we investigate associations between complex auditory and complex taste stimuli. A novel piece of music was composed and recorded in four different styles of musical articulation to reflect the four basic tastes groups (sweet, sour, salty, bitter). In Experiment 1, participants performed above chance at pairing the music clips with corresponding taste words. Experiment 2 uses multidimensional scaling to interpret how participants categorize these musical stimuli, and to show that auditory categories can be organized in a similar manner as taste categories. Experiment 3 introduces four different flavors of custom-made chocolate ganache and shows that participants can match music clips with the corresponding taste stimuli with above-chance accuracy. Experiment 4 demonstrates the partial role of pleasantness in crossmodal mappings between sound and taste. The present findings confirm that individuals are able to make crossmodal associations between complex auditory and gustatory stimuli, and that valence may mediate multisensory integration in the general population.

Cross-Modal Associations Between Color and Touch: Mapping Haptic and Tactile Terms to the Surface of the Munsell Color Solid

This study investigated cross-modal associations between color and touch using a matching task. Participants matched colors drawn from the surface of the Munsell color solid to antonym pairs of haptic/tactile adjectives. For most of the term pairs assessed (soft/hard, smooth/rough, flat/uneven, slippery/not slippery, light/heavy, thin/thick and round/sharp) matching appears predominantly influenced by lightness, with the first term from each pair matched to light colors and the other to dark colors, a result in close agreement with previous research. For two terms, warm and wet, there were clear influences of hue on task performance. There were also similarities between patterns of color matching to several of the haptic/tactile terms assessed and color matching to another term, dislike. This suggests valence may play a mediating role in cross-modal associations involving touch and color.

Emotional and Semantic Associations Between Cinematographic Aesthetics and Haptic Perception

This study investigates systematic links between haptic perception and multimodal cinema perception. It differs from previous research conducted on cross-modal associations as it focuses on a complex intermodal stimulus, close to one people experience in reality: cinema. Participants chose materials that are most/least consistent with three-minute samples of films with elements of beauty and ugliness. We found that specific materials are associated with certain films significantly different from chance. Silk was associated with films including elements of beauty, while sandpaper was associated with films including elements of ugliness. To investigate the nature of this phenomenon, we tested the mediation effect of emotional/semantic representations on cinema–haptic associations. We found that affective representations at least partly explain the cross-modal associations between films and materials.

Musical-Space Synaesthesia: VisualisationofMusicalTexture.

I, Svetlana Rudenko, am a concert pianist. I am also a synaesthete and for me, sound is visual. It has shape. According to recent research (Akiva-Kabiri et al., 2014, pp. 17-29): "In musical-space synaesthesia, …unlike the vertical and horizontal representation of musical pitch tones in the general population, synaesthetes describe a linear diagonal organisation of pitch tones." Different piano sounds have aroma and texture. The forms I have lived with since childhood are what I call 'sound landscapes'. I am not alone. Composers such as Liszt, Scriabin, Gubaidulina and Messiaen, as well as artists such as de Córdoba Serrano and Ninghui Xiong, also experienced these impressions. This paper will explore synaesthetic experiences of musical texture, visualisation of sound and tactile sensations of musical texture as well as enhanced cross-modal associations. On the basis of my experiences, I argue that the visualisation of musical texture influences sound perception and even timing: phrases, dynamics and the whole interpretation.

Modality-invariant audio-visual association coding in crow endbrain neurons

Single neuron activity in the corvid nidopallium caudolaterale (NCL), the supposed avian functional analog of the prefrontal cortex, represents associations of auditory with visual stimuli. This is of high adaptive value for songbirds that need to rely on audio-visual associations to communicate, find a mate or escape predators. However, it remains unclear whether NCL neurons can represent cross-modal associations in a modality invariant, abstract fashion. To dissociate between modality-dependent and modality-invariant NCL activity, we trained two crows to match auditory sample cues with visual test stimuli, and vice versa, across a temporal memory delay. During sample presentation, NCL activity selectively encoded associations in a modality invariant fashion. During the delay, we observed subject specific, population-level coding biases in NCL activity. Despite of these biases, task relevant information could be decoded equally well from either subject’s neuronal delay activity. Decoding success was facilitated by many mixed selectivity neurons, which mediated high dimensional representations of task variables on the NCL population level. These results parallel findings from the mammalian PFC, suggesting common mechanisms responsible for the adaptability of multimodal association areas across taxa.

Odor\u2013color associations differ with verbal descriptors for odors: A comparison of three linguistically diverse groups

People appear to have systematic associations between odors and colors. Previous research has emphasized the perceptual nature of these associations, but little attention has been paid to what role language might play. It is possible odor–color associations arise through a process of labeling; that is, participants select a descriptor for an odor and then choose a color accordingly (e.g., banana odor → “banana” label → yellow). If correct, this would predict odor–color associations would differ as odor descriptions differ. We compared speakers of Dutch (who overwhelmingly describe odors by referring to the source; e.g., smells like banana) with speakers of Maniq and Thai (who also describe odors with dedicated, abstract smell vocabulary; e.g., musty), and tested whether the type of descriptor mattered for odor–color associations. Participants were asked to select a color that they associated with an odor on two separate occasions (to test for consistency), and finally to label the odors. We found the hunter-gatherer Maniq showed few, if any, consistent or accurate odor–color associations. More importantly, we found the types of descriptors used to name the smells were related to the odor–color associations. When people used abstract smell terms to describe odors, they were less likely to choose a color match, but when they described an odor with a source-based term, their color choices more accurately reflected the odor source, particularly when the odor source was named correctly (e.g., banana odor → yellow). This suggests language is an important factor in odor–color cross-modal associations.

A pain in the bud? Implications of cross-modal sensitivity for pain experience.

There is growing evidence that enhanced sensitivity to painful clinical procedures and chronic pain are related to greater sensitivity to other sensory inputs, such as bitter taste. We examined cross-modal sensitivities in two studies. Study 1 assessed associations between bitter taste sensitivity, pain tolerance, and fear of pain in 48 healthy young adults. Participants were classified as non-tasters, tasters and super-tasters using a bitter taste test (6-n-propythiouracil; PROP). The latter group had significantly higher fear of pain (Fear of Pain Questionnaire) than tasters (p=.036, effect size r = .48). There was only a trend for an association between bitter taste intensity ratings and intensity of pain at the point of pain tolerance in a cold pressor test (p=.04). In Study 2, 40 healthy young adults completed the Adolescent/Adult Sensory Profile before rating intensity and unpleasantness of innocuous (33 °C), moderate (41 °C), and high intensity (44 °C) thermal pain stimulations. The sensory-sensitivity subscale was positively correlated with both intensity and unpleasantness ratings. Canonical correlation showed that only sensitivity to audition and touch (not taste/smell) were associated with intensity of moderate and high (not innocuous) thermal stimuli. Together these findings suggest that there are cross-modal associations predominantly between sensitivity to exteroceptive inputs (i.e., taste, touch, sound) and the affective dimensions of pain, including noxious heat and intolerable cold pain, in healthy adults. These cross-modal sensitivities may arise due to greater psychological aversion to salient sensations, or from shared neural circuitry for processing disparate sensory modalities.

Cross-modal association between auditory and visual-spatial information in Mandarin tone perception

Speech perception involves multiple input modalities. Research has indicated that perceivers may establish a cross-modal association between auditory and visual-spatial events to aid perception. Such intermodal relations can be particularly beneficial for non-native perceivers who need additional resources to process challenging new sounds. This study examines how co-speech hand gestures mimicking pitch contours in space affect non-native Mandarin tone perception. Native English as well as Mandarin perceivers identified tones with either congruent or incongruent auditory-facial and gestural (AF/G) input. Perceivers also identified congruent and incongruent auditory-facial (A/F) stimuli. Native Mandarin results showed the expected ceiling-level performance in the congruent A/F and AF/G conditions. In the incongruent conditions, while A/F identification was primarily auditory-based, AF/G identification was partially based on gestures, demonstrating the use of gestures as valid cues in tone identification. The English perceivers’ performance was poor in the congruent A/F condition, but improved significantly in AF/G. While the incongruent A/F identification showed some reliance on facial information, incongruent AF/G identification relied more on gestural than auditory-facial information. These results indicate positive effects of facial and especially gestural input on non-native tone perception, suggesting that cross-modal (visual-spatial) resources can be recruited to aid auditory perception when phonetic demands are high.