Cross-sensory Correspondences Research Articles

Paths of linguistic synesthesia across cultures

Abstract The article focuses on the conventionalized cross-sensory uses of basic-level adjectives in a sample of eight languages: English, German, French, Spanish, Russian, Hungarian, Tajik, and Uzbek. After a differentiation of cross-sensory language use (also called linguistic synesthesia) from other phenomena that combine the senses (namely, neuropsychological synesthesia and cross-sensory correspondences), it reports on a dictionary-based semantic analysis that distinguishes between three main semantic mechanisms leading to cross-sensory language use: direct cross-sensory transfer (e.g., a dark sound), more schematic generalized meanings (e.g., soft ‘pleasant, gentle, not too intense’), and highly figurative extensions (e.g., a dark melody, in which dark means ‘gloomy’). It also emphasizes that these three categories are often intertwined due to the inherent fog-like nature of meaning. After summarizing every instance of conventionalized cross-sensory meaning potential that could be found in the dictionaries, it concludes that (1) the results are in line with the widely observed directional preferences also referred to as the hierarchy of the senses; (2) the evaluative dimension is present in many transfers, but it cannot account for the extended uses alone; (3) there are some obvious differences between the Western and the Central Asian languages, even though one cannot speak of fundamentally different conceptual systems regarding the language of the senses. Besides these general observations, the outcomes of this principally exploratory investigation also point to many uncharted territories to be examined in future studies.

Heaviness-brightness correspondence and stimulus-response compatibility

Cross-sensory correspondences can reflect crosstalk between aligned conceptual feature dimensions, though uncertainty remains regarding the identities of all the dimensions involved. It is unclear, for example, if heaviness contributes to correspondences separately from size. Taking steps to dissociate variations in heaviness from variations in size, the question was asked if a heaviness-brightness correspondence will induce a congruity effect during the speeded brightness classification of simple visual stimuli. Participants classified the stimuli according to whether they were brighter or darker than the mid-gray background against which they appeared. They registered their speeded decisions by manipulating (e.g., tapping) the object they were holding in either their left or right hand (e.g., left for bright, right for dark). With these two otherwise identical objects contrasting in their weight, stimuli were classified more quickly when the relative heaviness of the object needing to be manipulated corresponded with the brightness of the stimulus being classified (e.g., the heavier object for a darker stimulus). This novel congruity effect, in the guise of a stimulus-response (S-R) compatibility effect, was induced when heaviness was isolated as an enduring feature of the object needing to be manipulated. It was also undiminished when participants completed a concurrent verbal memory load task, countering claims that the heaviness-brightness correspondence is verbally mediated. Heaviness, alongside size, appears to contribute to cross-sensory correspondences in its own right and in a manner confirming the far-reaching influence of correspondences, extending here to the fluency with which people communicate simple ideas by manipulating a hand-held object.

Sensus Communis: Some Perspectives on the Origins of Non-synchronous Cross-Sensory Associations.

Adults readily make associations between stimuli perceived consecutively through different sense modalities, such as shapes and sounds. Researchers have only recently begun to investigate such correspondences in infants but only a handful of studies have focused on infants less than a year old. Are infants able to make cross-sensory correspondences from birth? Do certain correspondences require extensive real-world experience? Some studies have shown that newborns are able to match stimuli perceived in different sense modalities. Yet, the origins and mechanisms underlying these abilities are unclear. The present paper explores these questions and reviews some hypotheses on the emergence and early development of cross-sensory associations and their possible links with language development. Indeed, if infants can perceive cross-sensory correspondences between events that share certain features but are not strictly contingent or co-located, one may posit that they are using a “sixth sense” in Aristotle’s sense of the term. And a likely candidate for explaining this mechanism, as Aristotle suggested, is movement.

Cross-sensory correspondences in language: Vowel sounds can symbolize the felt heaviness of objects.

In sound symbolism, a word's sound induces expectations about the nature of a salient aspect of the word's referent. P. Walker (2016a) proposed that cross-sensory correspondences can be the source of these expectations, and the present study assessed three implications flowing from this proposal. First, sound symbolism will embrace a wide range of referent features, including heaviness. Second, any feature of a word's sound able to symbolize one aspect of the word's referent will also be able to symbolize corresponding aspects of the referent (e.g., a sound feature symbolizing visual pointiness will also symbolize lightness in weight). Third, sound symbolism will be independent of the sensory modality through which a word's referent is encoded (e.g., whether heaviness is felt or seen). Adults judged which of two contrasting novel words was most appropriate as a name for the heavier or lighter of two otherwise identical hidden novel objects they were holding in their hands. The alternative words contrasted in their vowels and/or consonants, one or both of which were known to symbolize visual pointiness. Although the plosive or continuant nature of the consonants did not influence the judged appropriateness of a word to symbolize the heaviness of its referent, back/open vowels, compared to front/close vowels, were judged to symbolize felt heaviness. The symbolic potential of back/open vowels to represent felt heaviness, predicted on the basis of their symbolism of visual roundedness, supports the proposal that cross-sensory correspondences contribute to sound symbolism. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Newborns are sensitive to the correspondence between auditory pitch and visuospatial elevation.

Amodal (redundant) and arbitrary cross-sensory feature associations involve the context-insensitive mapping of absolute feature values across sensory domains. Cross-sensory associations of a different kind, known as correspondences, involve the context-sensitive mapping of relative feature values. Are such correspondences in place at birth (like amodal associations), or are they learned from subsequently experiencing relevant feature co-occurrences in the world (like arbitrary associations)? To decide between these two possibilities, human newborns (median age = 44 hr) watched animations in which two balls alternately rose and fell together in space. The pitch of an accompanying sound rose and fell either congruently with this visual change (pitch rising and falling as the balls moved up and down), or incongruently (pitch rising and falling as the balls moved down and up). Newborns' looking behavior was sensitive to this congruence, providing the strongest indication to date that cross-sensory correspondences can be in place at birth.

Cross-Sensory Correspondences: Heaviness is Dark and Low-Pitched.

Everyday language reveals how stimuli encoded in one sensory feature domain can possess qualities normally associated with a different domain (e.g., higher pitch sounds are bright, light in weight, sharp, and thin). Such cross-sensory associations appear to reflect crosstalk among aligned (corresponding) feature dimensions, including brightness, heaviness, and sharpness. Evidence for heaviness being one such dimension is very limited, with heaviness appearing primarily as a verbal associate of other feature contrasts (e.g., darker objects and lower pitch sounds are heavier than their opposites). Given the presumed bidirectionality of the crosstalk between corresponding dimensions, heaviness should itself induce the cross-sensory associations observed elsewhere, including with brightness and pitch. Taking care to dissociate effects arising from the size and mass of an object, this is confirmed. When hidden objects varying independently in size and mass are lifted, objects that feel heavier are judged to be darker and to make lower pitch sounds than objects feeling less heavy. These judgements track the changes in perceived heaviness induced by the size-weight illusion. The potential involvement of language, natural scene statistics, and Bayesian processes in correspondences, and the effects they induce, is considered.

Letter of the guest editors of the special issue on “Music as a multimodal experience”.

It is our pleasure to introduce a collection of papers brought together under the heading of as a Multimodal Experience. This issue aims to bring to the foreground recent developments in our understanding of music as a multimodal experience. Music listening generally concerns multiple modalities in the sense that other modalities than hearing are simultaneously present and influence how we perceive and engage with music. This multimodality includes visual aspects of music production or multimedia presentations of music, tactile experiences of frequency, intensity, and rhythm, kinematic responses to music, and so forth. However, even if we isolate the presentation mode of music to auditory, multiple sense-modalities are nevertheless involved in the perception of music, activated through cross-modal correspondences or cross-modal associations.To investigate the role of these cross-modal correspondences in the perception and cognition of music, a British Academy-funded network was established between researchers at The University of Sheffield, United Kingdom, Tel Aviv University, Israel, and The Hebrew University of Jerusalem, also in Israel. A number of research projects were initiated within this network, one of which is reported in this special issue (contribution by Timmers & Li). Additionally, the network organized the International Conference on the Multimodal Experience of Music, which was held March 23-25, 2015, at The University of Sheffield, United Kingdom. The papers brought together in this special issue provide a small, but relevant, cross-division of the wide ranging studies and topics presented at the conference (for further reading, see the ICMEM Proceedings)1.Firstly, Walker contextualizes research on cross-modal associations with musical parameters within a broader theoretical framework of cross-sensory correspondences. This is followed by four studies that investigate cross-modal associations that may occur while listening to music. The contribution by Miller explores the presence of pitch-related kinesthetic vocal sensations during music listening (i.e., in the vocal tract, ears, eustachian tubes), examining the hypothesis that auditory and vocal processes share peripheral substrates. The study by Metcalfe investigates the relationship between musical attributes (tempo, intensity, and jitter in periodicity) and spontaneous adaptations in walking when listening to the music. It suggests an integrative model in which both arousal and rhythmic entrainment interact in modulating walking speed. Timmers and Li investigate factors contributing to a cross-modally induced perceptual illusion: The perception of the horizontal location of brief sounds is influenced by the pitch of the sound. An illusion of high sounds coming from the right and low sounds coming from the leftis stronger for participants with higher levels of musical training. It is also stronger for pianist than for flutists, however, only after performing their respective instrument. Isben and Krumhansl explore the possible mediating role of the perception of emotion in music for music-color associations. They present clear analogies between attributed emotions to colors, attributed emotions to music, and attributed colors to music, which is further supported by predicting music-color associations based on color- emotion mappings.The second set of studies presented in the issue examine the contribution of different modalities to the perception of music presented in a unimodal versus a multimodal fashion. Slater and Marozeau investigate whether the addition of tactile cues can facilitate the segregation of a melody within a context in which two melodies are interleaved and (without assistance for selective attention) perceptually integrated. …

Cross-sensory correspondences: A theoretical framework and their relevance to music.

Each of our senses is 'blind' to some features of objects and events (e.g., hearing can tell us little or nothing about the shape, colour, and weight of an object, or about how it might taste or smell). When we listen to sounds without support from other sensory modalities, such as when listening to recorded music, how do will fill-in these blind spots? Evidence identifying a core set of cross-sensory correspondences among basic stimulus features is reviewed, and it is proposed that they offer a potential basis for the filling-in of information that is missing when one or more sensory systems is not available. An emerging theoretical framework for understanding correspondences and their impact on behaviour is presented. Evidence pertaining to key features of the framework is reviewed, including that cross-sensory correspondences are based on cross-talk among conceptual representations of aligned feature dimensions, are bi-directional in their effects, obey transitivity in the feature associations they support, involve the relative (context-sensitive) coding of stimulus features, and can be accessed through the verbal specification of feature values. After illustrating how cross-sensory correspondences are able to embrace basic features of bodily actions, gestures, and vocalisations, their potential for exploitation in the communication of ideas is explained. The relevance of cross-sensory correspondences to musical sounds, and their potential to enhance the composition, performance, and appreciation of music, are discussed.

Cross-sensory mapping of feature values in the size-brightness correspondence can be more relative than absolute.

A role for conceptual representations in cross-sensory correspondences has been linked to the relative (context-sensitive) mapping of feature values, whereas a role for sensory-perceptual representations has been linked to their absolute (context-insensitive) mapping. Demonstrating the relative nature of the automatic mapping underlying a cross-sensory correspondence therefore offers one way of confirming its conceptual basis. After identifying several prerequisites for relative and absolute mappings, we provide the first compelling demonstration that an automatically induced congruity effect based on a cross-sensory correspondence (i.e., that between haptic size and visual brightness) can be largely contingent on the relative mapping of the 2 features, thereby implying a conceptual basis for the correspondence. Participants in a speeded classification task were faster to classify a visual stimulus as brighter or darker when this required them to press a hidden response key that, incidentally, was relatively small or big, respectively. Importantly, the same levels of brightness (Experiment 1) and key size (Experiment 2) at different times corresponded to contrasting levels of the other feature depending on the context provided by the alternative stimuli with which they appeared. For example, the same medium key was congruent with a brighter stimulus when paired with a bigger key, but was congruent with a darker stimulus when paired with a smaller key. Reflecting on the broader implications of this finding, it is noted that the involvement of cross-sensory correspondences in some forms of sound symbolism in language also requires the relative coding of stimulus features.

Cross-sensory correspondences and symbolism in spoken and written language.

Lexical sound symbolism in language appears to exploit the feature associations embedded in cross-sensory correspondences. For example, words incorporating relatively high acoustic frequencies (i.e., front/close rather than back/open vowels) are deemed more appropriate as names for concepts associated with brightness, lightness in weight, sharpness, smallness, speed, and thinness, because higher pitched sounds appear to have these cross-sensory features. Correspondences also support prosodic sound symbolism. For example, speakers might raise the fundamental frequency of their voice to emphasize the smallness of the concept they are naming. The conceptual nature of correspondences and their functional bidirectionality indicate they should also support other types of symbolism, including a visual equivalent of prosodic sound symbolism. For example, the correspondence between auditory pitch and visual thinness predicts that a typeface with relatively thin letter strokes will reinforce a word's reference to a relatively high pitch sound (e.g., squeal). An initial rating study confirms that the thinness-thickness of a typeface's letter strokes accesses the same cross-sensory correspondences observed elsewhere. A series of speeded word classification experiments then confirms that the thinness-thickness of letter strokes can facilitate a reader's comprehension of the pitch of a sound named by a word (thinner letter strokes being appropriate for higher pitch sounds), as can the brightness of the text (e.g., white-on-gray text being appropriate for the names of higher pitch sounds). It is proposed that the elementary visual features of text are represented in the same conceptual system as word meaning, allowing cross-sensory correspondences to support visual symbolism in language. (PsycINFO Database Record

The color of music: Emotion-mediated associations to Bach’s Well-tempered Clavier.

Associations between music and color, and cross-modal associations in general, have been understood in a variety of ways. One view is that, because we experience the world through multiple sensory modalities simultaneously, integrating these signals provides us with richer and more precise information on which to act (Parise & Spence, 2013; Spence, 2011). Cross-modal associations, for example, between the color of a fruit and whether or not it tastes ripe, are normal and adaptive for integrating information across the senses. Three types of cross-modal associations that have been identified are structural correspondences (originating in the neural processing), statistical correspondences (reflecting regularities in the world), and semantically (primarily verbally) mediated correspondences (Spence, 2011).Walker and colleagues (Walker & Smith, 1984; Walker, Walker, & Francis, 2012) have proposed a system of amodal, conceptual terms that apply across sensory modalities to account for cross-modal associations. These draw on basic features of bodily actions, gestures, and vocalizations. For example, common to hearing and sight are fast versus slow, high versus low, bright versus dark, and small versus large. Such cross-sensory correspondences may be bidirectional and symmetrical, or asymmetrical with the association in one direction being stronger than in the other, or even unidirectional. Hearing a high sound might lead one to expect that a small object is the source, but a small object would not necessarily lead one to expect a high sound. Walker (2016) describes how the theory might apply to music, going beyond associations to simple tones varying in pitch height, loudness, and/or timbre, to possible factors operating at compositional levels of melody and tempo.One issue that arises in the literature on cross-modal associations is what the relationship is between these associations and the phenomenon of synesthesia, in which a trigger stimulus (inducer) causes automatic, perception-like experiences (concurrent) in a different modality. Although cross-modal associations have been found to be consistent across individuals, the focus in research on synesthesia tends to be on individuals under the assumption that the particular correspondences would be idiosyncratic. Hundreds of different types of synesthesia have been identified, with a history of scientific reports going back to the 1800s (Galton, 1883; see Jewanski, 2013; Ward, 2013 for reviews). Recently, it has been suggested that synesthesia may be subject to the same rules that govern normal patterns of cross-modal associations, differing mainly in its automaticity but less in its phenomenology (Marks, 2013; Simner, 2013; but see Spence, 2011).Marks (1975) reviewed >100 reports of auditory-visual synesthesia, many of which involved music. He found that some of the associations are regular, systematic, and consistent across individuals. Specifically, there were correlations between auditory pitch and visual brightness (the higher the brighter, and in some studies more yellow), and pitch and visual size (larger with lower), and loudness and visual size and brightness (larger and brighter with louder). Marks described devices invented to present colors to accompany music, such as an organ built by Scriabin that mapped tones and colors so that, for example, C was accompanied by red, and D accompanied by yellow. Associations between musical notes and/or keys and colors are quite common in synesthetes and a case study demonstrated stability over time (Carroll & Greenberg, 1961). However, as noted by Marks, the specific mappings tend to vary across individuals, so that C might be red to one synesthete and white to another. Other, also idiosyncratic, associations have been reported between color and musical instruments, such as trumpet and scarlet, and flute and blue, and the music of particular composers, such as Chopin and purple, Mozart and green, or specific works by composers. …

The size-brightness correspondence: evidence for crosstalk among aligned conceptual feature dimensions.

The same core set of cross-sensory correspondences connecting stimulus features across different sensory channels are observed, regardless of the modality of the stimulus with which the correspondences are probed. This observation suggests that correspondences involve modality-independent representations of aligned conceptual feature dimensions and predicts a size-brightness correspondence in which smaller is aligned with brighter. This suggestion accommodates cross-sensory congruity effects where contrasting feature values are specified verbally rather than perceptually (e.g., where the words WHITE and BLACK interact with the classification of high- and low-pitch sounds). Experiment 1 brings these two issues together in assessing a conceptual basis for correspondences. The names of bright/white and dark/black substances were presented in a speeded brightness classification task, in which the two alternative response keys differed in size. A size-brightness congruity effect was confirmed, with substance names classified more quickly when the relative size of the response key needing to be pressed was congruent with the brightness of the named substance (e.g., when yoghurt was classified as a bright substance by pressing the smaller of two keys). Experiment 2 assesses the proposed conceptual basis for this congruity effect by requiring the same named substances to be classified according to their edibility (with all of the bright/dark substances having been selected for their edibility/inedibility, respectively). The predicted absence of a size-brightness congruity effect, along with other aspects of the results, supports the proposed conceptual basis for correspondences and speaks against accounts in which modality-specific perceptuomotor representations are entirely responsible for correspondence-induced congruity effects.

Cross-sensory correspondences and cross talk between dimensions of connotative meaning: Visual angularity is hard, high-pitched, and bright

Higher-pitched sounds are judged to be, among other things, sharper, harder, and brighter than lower-pitched sounds. Following Karwoski, Odbert, and Osgood (Journal of General Psychology 26:199-222, 1942), such cross-sensory correspondences are proposed to have a semantic basis, reflecting extensive bidirectional cross-activation among dimensions of connotative meaning. On this basis, the same core set of correspondences should emerge whichever sensory feature is used to probe it. More angular (sharper) shapes should, for example, be higher-pitched and have the same cross-sensory features as higher-pitched sounds. Experiments 1-3 employed a speeded classification task designed to reveal cross-sensory correspondences having a semantic basis. With words as to-be-classified stimuli and with shapes varying in angularity as concurrent incidental stimuli, congruity effects between angularity and each of hardness, pitch, and brightness were confirmed. Correspondences with a semantic basis need not be cross-modality in nature. Experiment 4 confirmed this by reproducing the brightness-angularity congruity effect when contrasting values for both features were encoded nonverbally within the visual modality. The varying nature and origins of cross-sensory correspondences and the basis on which they induce congruity effects in speeded classification are explored.

Size–brightness correspondence: Crosstalk and congruity among dimensions of connotative meaning

Following Karwoski, Odbert, and Osgood (Journal of General Psychology 26:199-222, 1942), in the present article, cross-sensory correspondences are proposed to reflect the alignment of, and extensive bidirectional cross-activation among, dimensions of connotative meaning. The size-brightness correspondence predicted on this basis (in which smaller is aligned with brighter) was confirmed in two ways. First, when participants explored three wooden balls of different size by touch alone and indicated how bright they thought each of them was, the smaller ball was judged to be brighter than the bigger ball. Second, when these two balls served as response keys in a speeded brightness-classification task, participants were quicker and more likely to be correct when confirming that a stimulus was bright (dark) when this required them to press the smaller (bigger) key, than when it required them to press the bigger (smaller) key. This congruity effect originated from interactions embedded in the later stages of information processing concerned with stimulus classification and response selection. These results, together with the observation that the cross-sensory features associated with smallness are the same as those associated with higher pitch sounds (i.e., both attributes are more active, brighter, faster, lighter in weight, quieter, sharper, and weaker than their opposites), support the suggestion that there exists a core set of cross-sensory correspondences that emerges whichever stimulus feature is used to probe it.

A Common Scheme for Cross-Sensory Correspondences across Stimulus Domains

Following Karwoski, Odbert, and Osgood (1942, Journal of General Psychology 26 199-222), it is proposed that cross-sensory correspondences can arise from extensive, bi-directional cross-activation between dimensions of connotative meaning. If this account is correct, the same set of cross-sensory correspondences (eg brightness with high pitch, high pitch with sharpness, smallness with brightness) should emerge regardless of the sensory channel (auditory, visual, or tactile) that is probed. To test this prediction, participants rated a range of auditory, visual, and tactile stimuli on a series of rating scales relating to different dimensions of connotative meaning. With only a few minor exceptions, the same set of cross-sensory correspondences emerged from all types of stimulus variation. This supports the suggestion that cross-sensory correspondences can reflect reciprocal interactions between dimensions of connotative meaning, and indicates that Spence's (2011, Attention, Perception, & Psychophysics 73 971-995) theoretical framework might be usefully extended to include semantically-based correspondences.

A common scheme for cross-sensory correspondences across stimulus domains

Following Karwoski et al. (1942), it is proposed that cross-sensory correspondences can arise from extensive, bidirectional cross-activation between dimensions of connotative meaning. If this account is correct, the same set of cross-sensory correspondences (e.g., smallness with brightness, brightness with high pitch, high pitch with sharpness) should emerge regardless of the sensory channel (visual, auditory or tactile) that is probed. To test this prediction, participants rated a range of visual, auditory, and tactile stimuli on a series of rating scales relating to different dimensions of connotative meaning. The same set of cross-sensory correspondences emerged from all types of stimulus variation. This supports the suggestion that cross-sensory correspondences can reflect reciprocal interactions between dimensions of connotative meaning, and indicates that Spence’s (2011) theoretical framework might be usefully extended to include semantically-based correspondences.

Size–brightness correspondence: Crosstalk and congruity among dimensions of connotative meaning

Using a speeded classification task, Walker and Walker (2012) demonstrated a cross-sensory correspondence between haptic size and surface brightness. Specifically, adult participants classified bright (dark) visual stimuli more quickly and accurately when this required them to press the smaller (bigger) of two response keys which were always hidden from view. The nature of the correspondence (i.e., small being aligned with bright), along with various aspects of the task situation, indicated that the congruity effect originated at later stages of information processing concerned with the semantic classification of stimuli and response selection. The study reported here provides additional evidence for the involvement of semantic coding. When the names of bright (white) edible substances (e.g., flour) and dark (black) inedible substances (e.g., soot) were classified according to their surface brightness, the same size–brightness congruity effect was observed. However, when the basis for classification of the substances was switched to their edibility, the congruity effect disappeared. It is therefore proposed that congruity effects based on cross-sensory correspondences can reflect interactions between the connotative meanings of elementary stimulus features (cf. Karwoski et al., 1942).

Cross-Sensory Correspondences and Naive Conceptions of Natural Phenomena

Cross-sensory correspondences automatically intrude on performance in elaborate laboratory tasks (see Spence 2011 Attention, Perception, & Psychophysics 73 971-995, for a review). Outside such tasks, might they be responsible for some popular misconceptions about natural phenomena? Four simple demonstrations reveal how the correspondences between surface-lightness and weight, and between surface-lightness and auditory pitch, generate misconceptions about the weight and movement of objects and the vocalisations of animals. Specifically, people expect darker objects to be heavier than lighter-coloured objects, to free-fall more quickly, to roll across a table more slowly, and to make lower-pitched vocalisations when they come to life.

Synesthesia: Strong and Weak

In this review, we distinguish strong and weak forms of synesthesia. Strong synesthesia is characterized by a vivid image in one sensory modality in response to stimulation in another one. Weak synesthesia is characterized by cross-sensory correspondences expressed through language, perceptual similarity, and perceptual interactions during information processing. Despite important phenomenological dissimilarities between strong and weak synesthesia, we maintain that the two forms draw on similar underlying mechanisms. The study of strong and weak synesthetic phenomena provides an opportunity to enrich scientists' understanding of basic mechanisms involved in perceptual coding and cross-modal information processing.