Cross-modal Associations Research Articles

The Framework and Memristive Circuit Design for Multisensory Mutual Associative Memory Networks.

In this work, we propose a multisensory mutual associative memory networks framework and memristive circuit to mimic the ability of the biological brain to make associations of information received simultaneously. The circuit inspired by neural mechanisms of associative memory cells mainly consists of three modules: 1) the storage neurons module, which encodes external multimodal information into the firing rate of spikes; 2) the synapse module, which uses the nonvolatility memristor to achieve weight adjustment and associative learning; and 3) the retrieval neuron module, which feeds the retrieval signal output from each sensory pathway to other sensory pathways, so that achieve mutual association and retrieval between multiple modalities. Different from other one-to-one or many-to-one unidirectional associative memory work, this circuit achieves bidirectional association from multiple modalities to multiple modalities. In addition, we simulate the acquisition, extinction, recovery, transmission, and consolidation properties of associative memory. The circuit is applied to cross-modal association of image and audio recognition results, and episodic memory is simulated, where multiple images in a scene are intramodal associated. With power and area analysis, the circuit is validated as hardware-friendly. Further research to extend this work into large-scale associative memory networks, combined with visual-auditory-tactile-gustatory sensory sensors, is promising for application in intelligent robotic platforms to facilitate the development of neuromorphic systems and brain-like intelligence.

Reality bites: highlighting the potential discrepancies between multisensory taste perception in extended and physical reality

IntroductionWhen we taste, we take in a variety of sensory information that can be completely independent from the properties of the food itself: the ambient lighting and environmental sounds can all influence our taste perception and affective responses. However, current multisensory research is mixed as to whether these In Real Life (IRL) findings also apply to Extended Reality (XR) environments. A potential reason for this may be the limited realism of some XR scenarios, which this study aimed to overcome through an immersive Virtual Reality experience (VR, chosen for its greater realism relative to other XR applications) whilst also expanding the scope to flavour perception.MethodsA total of 34 participants rated food samples under neutral, red, and green ambient lighting in VR. Participants ate either lime flavoured, strawberry flavoured, or “Neutral” (no added flavour) samples.ResultsWhile participants were equally immersed in all three environments, they rated the red and green lighting environments as substantially less natural than the neutral lighting environment. Interestingly, while participants associated sweetness and sourness with red lighting and green lighting respectively, this minimally extended to their behavioural ratings, when sampling the stimuli in VR. Samples eaten under red lighting were rated as significantly sweeter than those eaten under neutral lighting. However, neither red nor green lighting affected sample sourness nor the intensity of strawberry or lime flavour.DiscussionThis study found limited evidence of multisensory integration in XR, wherein taste expectations do not extend to taste experiences. We discuss these results in light of prior works on crossmodal associations and raise a reflection on why multisensory integration of taste may not apply to XR in our study. While there is much excitement about the opportunities XR can offer, we argue that we are only at the beginning of understanding the experiences on the reality-virtuality continuum and that we need to establish a richer understanding of participants' experiences, expectations, and taste/flavour perceptions between IRL and XR.

Color-temperature Cross-modal Association: The Relationships of Hue, Saturation, and Brightness with Temperature

Color-temperature Cross-modal Association: The Relationships of Hue, Saturation, and Brightness with Temperature

Vitamin D deficiency associated with neurodevelopmental problems in 2-year-old Japanese boys.

While associations between vitamin D deficiency and neurodevelopmental disorders have been found, large studies on child vitamin D, neurodevelopment, and sex differences among the general population are lacking. This study aimed to investigate the association between child serum 25-hydroxyvitamin D (25(OH)D)) levels and neurodevelopmental problems (NDPs). Serum 25(OH)D and NDPs were measured at age two among the subcohort study of the Japan Environment and Children's Study. NDPs were assessed with the Kyoto Scale of Psychological Development 2001 (Kyoto scale). Adjusted odds ratios (aORs) for the Kyoto-scale developmental quotient scores <70 were calculated, for postural-motor, cognitive-adaptive, and language-social domains and overall scores, adjusted for test month, latitude, small for gestational age, maternal age, and daycare attendance. Among 2363 boys and 2290 girls, boys had higher 25(OH)D levels, but scored lower in the Kyoto scale. For boys in the vitamin D deficiency (<20 ng/mL) group, aORs of scoring the Kyoto-scale DQs <70 were 2.33 (p = 0.006) for overall DQs, 1.91 (p = 0.037) for cognitive-adaptive, and 1.69 (p = 0.024) for language-social domains. For girls, results were inconclusive. Only boys showed a clear and cross-modal association between vitamin D deficiency and NDPs.

Sensory translation between audition and vision.

Across the millennia, and across a range of disciplines, there has been a widespread desire to connect, or translate between, the senses in a manner that is meaningful, rather than arbitrary. Early examples were often inspired by the vivid, yet mostly idiosyncratic, crossmodal matches expressed by synaesthetes, often exploited for aesthetic purposes by writers, artists, and composers. A separate approach comes from those academic commentators who have attempted to translate between structurally similar dimensions of perceptual experience (such as pitch and colour). However, neither approach has succeeded in delivering consensually agreed crossmodal matches. As such, an alternative approach to sensory translation is needed. In this narrative historical review, focusing on the translation between audition and vision, we attempt to shed light on the topic by addressing the following three questions: (1) How is the topic of sensory translation related to synaesthesia, multisensory integration, and crossmodal associations? (2) Are there common processing mechanisms across the senses that can help to guarantee the success of sensory translation, or, rather, is mapping among the senses mediated by allegedly universal (e.g., amodal) stimulus dimensions? (3) Is the term 'translation' in the context of cross-sensory mappings used metaphorically or literally? Given the general mechanisms and concepts discussed throughout the review, the answers we come to regarding the nature of audio-visual translation are likely to apply to the translation between other perhaps less-frequently studied modality pairings as well.

Dual-Path Rare Content Enhancement Network for Image and Text Matching

Image and text matching plays a crucial role in bridging the cross-modal gap between vision and language, and has achieved great progress due to the deep learning. However, the existing methods still suffer from the long-tail problem, where only a small proportion contains highly frequent semantics and a long tail proportion is constructed by rare semantics. In this paper, we propose a novel Dual-path Rare Content Enhancement Network (DRCE) to tackle the long-tail issue. Specifically, the Cross-modal Representation Enhancement (CRE) and Cross-modal Association Enhancement (CAE) are proposed to construct dual-path structure to enhance rare content representation and association with the benefit of cross-modal prior knowledge. This structure can effectively exploit the complementary cross-modal relation from different aspects and fuse these information in an adaptively manner by the proposed Adaptive Fusion Strategy (AFS). Moreover, we also propose an alternative re-ranking strategy (ARR) to explore the reciprocal contextual information to refine image-text matching results, which can further suppress the negative effect of long-tail effect. Extensive experiments on two large-scale datasets show the significant improvements and validate the superiority of our method.

Perceptual Similarity: Insights From Crossmodal Correspondences

AbstractPerceptual similarity is one of the most fiercely debated topics in the philosophy and psychology of perception. The documented history of the issue spans all the way from Plato – who regarded similarity as a key factor for human perceptual experience and cognition – through to contemporary psychologists – who have tried to determine whether, and if so, how similarity relationships can be established between stimuli both within and across the senses. Recent research on cross-sensory associations, otherwise known as crossmodal correspondences – that is, the existence of observable consensual associations, or mappings, between stimuli across different senses – represents an especially interesting field in which to study perceptual similarity. In fact, most accounts of crossmodal association that have been put forward in the literature to date evoke perceptual similarity as a key explanatory factor mediating the underlying association. At the same time, however, these various accounts raise several important theoretical questions concerning the very nature of similarity, with, for example, the sensory, affective, or cognitive underpinnings of similarity judgements remaining unclear. We attempt to shed light on these questions by examining the various accounts of crossmodal associations that have been put forward in the literature. Our suggestion is that perceptual similarity varies from being phenomenologically-based to conceptually-based. In particular, we propose that the nature of the associations underlying similarity judgements – whether these associations are phenomenologically-, structurally-, emotionally-, or conceptually-based – may be represented in a two-dimensional space with associative strength on one axis, and cognitive penetrability on the other.

Automatic Lung Cancer Subtypes Classification on CT Images with Self-generated Multi-modality Hybrid Features.

Lung cancer is a malignant tumor with rapid progression and high fatality rate. According to histological morphology and cell behaviours of cancerous tissues, lung cancer can be classified into a variety of subtypes. Since different cancer subtype corresponds to distinct therapies, the early and accurate diagnosis is critical for following treatments and prognostic managements. In clinical practice, the pathological examination is regarded as the gold standard for cancer subtypes diagnosis, while the disadvantage of invasiveness limits its extensive use, leading the non-invasive and fast-imaging computed tomography (CT) test a more commonly used modality in early cancer diagnosis. However, the diagnostic results of CT test are less accurate due to the relatively low image resolution and the atypical manifestations of cancer subtypes. In this work, we propose a novel automatic classification model to offer the assistance in accurately diagnosing the lung cancer subtypes on CT images. Inspired by the findings of cross-modality associations between CT images and their corresponding pathological images, our proposed model is developed to incorporate general histopathological information into CT imagery-based lung cancer subtypes diagnostic by omitting the invasive tissue sample collection or biopsy, and thereby augmenting the diagnostic accuracy. Experimental results on both internal evaluation datasets and external evaluation datasets demonstrate that our proposed model outputs more accurate lung cancer subtypes diagnostic predictions compared to existing CT-based state-of-the-art (SOTA) classification models, by achieving significant improvements in both accuracy (ACC) and area under the receiver operating characteristic curve (AUC).Clinical Relevance- This work provides a method for automatically classifying the lung cancer subtypes on CT images.

Crossmodal association between visual and acoustic cues in a tortoise (Testudo hermanni).

Humans spontaneously match information coming from different senses, in what we call crossmodal associations. For instance, high-pitched sounds are preferentially associated with small objects, and low-pitched sounds with larger ones. Although previous studies reported crossmodal associations in mammalian species, evidence for other taxa is scarce, hindering an evolutionary understanding of this phenomenon. Here, we provide evidence of pitch-size correspondence in a reptile, the tortoise Testudo hermanni. Tortoises showed a spontaneous preference to associate a small disc (i.e. visual information about size) with a high-pitch sound (i.e. auditory information) and a larger disc to a low-pitched sound. These results suggest that crossmodal associations may be an evolutionary ancient phenomenon, potentially an organizing principle of the vertebrate brain.

Gender congruence and emotion effects in cross-modal associative learning: Insights from ERPs and pupillary responses.

Social and emotional cues from faces and voices are highly relevant and have been reliably demonstrated to attract attention involuntarily. However, there are mixed findings as to which degree associating emotional valence to faces occurs automatically. In the present study, we tested whether inherently neutral faces gain additional relevance by being conditioned with either positive, negative, or neutral vocal affect bursts. During learning, participants performed a gender-matching task on face-voice pairs without explicit emotion judgments of the voices. In the test session on a subsequent day, only the previously associated faces were presented and had to be categorized regarding gender. We analyzed event-related potentials (ERPs), pupil diameter, and response times (RTs) of N = 32 subjects. Emotion effects were found in auditory ERPs and RTs during the learning session, suggesting that task-irrelevant emotion was automatically processed. However, ERPs time-locked to the conditioned faces were mainly modulated by the task-relevant information, that is, the gender congruence of the face and voice, but not by emotion. Importantly, these ERP and RT effects of learned congruence were not limited to learning but extended to the test session, that is, after removing the auditory stimuli. These findings indicate successful associative learning in our paradigm, but it did not extend to the task-irrelevant dimension of emotional relevance. Therefore, cross-modal associations of emotional relevance may not be completely automatic, even though the emotion was processed in the voice.

Motivated attention and task relevance in the processing of cross-modally associated faces: Behavioral and electrophysiological evidence

It has repeatedly been shown that visually presented stimuli can gain additional relevance by their association with affective stimuli. Studies have shown effects of associated affect in event-related potentials (ERP) like the early posterior negativity (EPN), late positive complex (LPC), and even earlier components as the P1 or N170. However, findings are mixed as to the extent associated affect requires directed attention to the emotional quality of a stimulus and which ERP components are sensitive to task instructions during retrieval. In this preregistered study (https://osf.io/ts4pb), we tested cross-modal associations of vocal affect-bursts (positive, negative, neutral) to faces displaying neutral expressions in a flash-card-like learning task, in which participants studied face-voice pairs and learned to correctly assign them to each other. In the subsequent EEG test session, we applied both an implicit (“old-new”) and explicit (“valence-classification”) task to investigate whether the behavior at retrieval and neurophysiological activation of the affect-based associations were dependent on the type of motivated attention. We collected behavioral and neurophysiological data from 40 participants who reached the preregistered learning criterium. Results showed EPN effects of associated negative valence after learning and independent of the task. In contrast, modulations of later stages (LPC) by positive and negative associated valence were restricted to the explicit, i.e., valence-classification, task. These findings highlight the importance of the task at different processing stages and show that cross-modal affect can successfully be associated to faces.

Network Localization of Awareness in Visual and Motor Anosognosia.

Unawareness of a deficit, anosognosia, can occur for visual or motor deficits and lends insight into awareness itself; however, lesions associated with anosognosia occur in many different brain locations. We analyzed 267 lesion locations associated with either vision loss (with and without awareness) or weakness (with and without awareness). The network of brain regions connected to each lesion location was computed using resting-state functional connectivity from 1,000 healthy subjects. Both domain specific and cross-modal associations with awareness were identified. The domain-specific network for visual anosognosia demonstrated connectivity to visual association cortex and posterior cingulate while motor anosognosia was defined by insula, supplementary motor area, and anterior cingulate connectivity. A cross-modal anosognosia network was defined by connectivity to the hippocampus and precuneus (false discovery rate p < 0.05). Our results identify distinct network connections associated with visual and motor anosognosia and a shared, cross-modal network for awareness of deficits centered on memory-related brain structures. ANN NEUROL 2023;94:434-441.

Assessing mechanisms behind crossmodal associations between visual textures and temperature concepts.

In the last decades, there has been a growing interest in crossmodal correspondences, including those involving temperature. However, only a few studies have explicitly examined the underlying mechanisms behind temperature-related correspondences. Here, we investigated the relative roles of an underlying affective mechanism and a semantic path (i.e., regarding the semantic knowledge related to a single common source identity or meaning) in crossmodal associations between visual textures and temperature concepts using an associative learning paradigm. Two online experiments using visual textures previously shown to be associated with low and high thermal effusivity (Experiment 1) and visual textures with no consensual associations with thermal effusivity (Experiment 2) were conducted. Participants completed a speeded categorization task before and after an associative learning task, in which they learned mappings between the visual textures and specific affective or semantic stimuli related to low and high temperatures. Across the two experiments, both the affective and semantic mappings influenced the categorization of visual textures with the hypothesized temperatures, but there was no influence on the reaction times. The effect of learning semantic mappings was larger than that of affective ones in both experiments, suggesting that a semantic path has more weight than an affective mechanism in the formation of the associations studied here. The associations studied here could be modified through associative learning establishing correlations between visual textures and either affective or semantic stimuli. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

The Application of Synesthetic Principles to Foster Musical Creativity

In 2014, the National Association for Music Education released music standards that strongly emphasized creativity, encouraging teachers to find new ways to implement creative activities into their classrooms. As such, making cross-modal associations and emphasizing divergent thinking are important priorities for teachers to consider. Synesthesia involves involuntary cognitive connections between senses (e.g., color and sound) that many people experience. The principles of this condition can be applied to all music learners, and building instructional approaches from them could help students make these unique associations and use divergent thinking to ultimately become more creative music makers.

Impacts of sensory multimodality congruence and familiarity with short use on cosmetic product evaluation.

Cross-modal association between sensory modalities is a natural phenomenon in the perception of our environment. For cosmetic evaluation, touch and smell are the two major sensory modalities involved in the whole product perception. In this study, we investigate whether a specific cosmetic texture is preferentially associated with a specific fragrance: congruence between texture and fragrance. In addition, we investigate whether 1-week use of a fragrance-texture congruent or non-congruent product can influence user's whole product appreciation and well-being. We have conducted a four-test experiment with 29 participants; first in the laboratory to evaluate: six fragrances and four textures individually with free description (test 1); the same stimuli with a description with cross-modal descriptors (test 2); 10 fragrance-texture combined products (test 3); and secondly at home, two fragrance-texture combined products: one congruent and one non-congruent (test 4). Results showed that: (1) For a given texture type, specific olfactory notes are necessary to lead to a congruent cross-modal pairing product. (2) Sensory modal congruent products produce the highest hedonic response. (3)Real-life use or familiarisation with a product can influence not only the degree of cross-modal congruence but also overall cosmetic product appreciation.

Steady state visual evoked potentials reveal a signature of the pitch-size crossmodal association in visual cortex

Crossmodal correspondences describe our tendency to associate sensory features from different modalities with each other, such as the pitch of a sound with the size of a visual object. While such crossmodal correspondences (or associations) are described in many behavioural studies their neurophysiological correlates remain unclear. Under the current working model of multisensory perception both a low- and a high-level account seem plausible. That is, the neurophysiological processes shaping these associations could commence in low-level sensory regions, or may predominantly emerge in high-level association regions of semantic and object identification networks. We exploited steady-state visual evoked potentials (SSVEP) to directly probe this question, focusing on the associations between pitch and the visual features of size, hue or chromatic saturation. We found that SSVEPs over occipital regions are sensitive to the congruency between pitch and size, and a source analysis pointed to an origin around primary visual cortices. We speculate that this signature of the pitch-size association in low-level visual cortices reflects the successful pairing of congruent visual and acoustic object properties and may contribute to establishing causal relations between multisensory objects. Besides this, our study also provides a paradigm can be exploited to study other crossmodal associations involving visual stimuli in the future.

MM-ConvBERT-LMS: Detecting Malicious Web Pages via Multi-Modal Learning and Pre-Trained Model

In recent years, the number of malicious web pages has increased dramatically, posing a great challenge to network security. While current machine learning-based detection methods have emerged as a promising alternative to traditional detection techniques. However, these methods are commonly based on single-modal features or simple stacking of classifiers built on various features. As a result, these techniques are not capable of effectively fusing features from different modalities, ultimately limiting the detection effectiveness. To address this limitation, we propose a malicious web page detection method based on multi-modal learning and pre-trained models. First, in the input stage, the raw URL and HTML tag sequences of web pages are used as input features. To help the subsequent model learn the relationship between the two modalities and avoid information confusion, modal-type encoding, and positional encoding are introduced. Next, a single-stream neural network based on the ConvBERT pre-trained model is used as the backbone classifier, and it learns the representation of multi-modal features through fine-tuning. For the output part of the model, a linear layer based on large margin softmax is applied to the decision-making. This activation function effectively increases the classification boundary and improves the robustness. In addition, a coarse-grained modal matching loss is added to the model optimization objective to assist the models in learning the cross-modal association features. Experimental results on synthetic datasets show that our proposed method outperforms traditional single-modal detection methods in general, and has advantages over baseline models in terms of accuracy and reliability.

The Nature of Paired Associate Learning Deficits in Chinese Children with Developmental Dyslexia

Previous studies have found that individuals with dyslexia perform poorly in paired associate learning (PAL) tasks, which were explained by a deficit in cross-modal association or verbal demand in alphabetic language. However, the nature of PAL deficits in non-alphabetic languages remains unclear. In this study, we conducted PAL and priming tasks in visual-visual, visual-verbal, verbal-visual, and verbal-verbal conditions to dissociate the cross-modal and verbal demands in Chinese children with dyslexia. In Experiment 1, children with dyslexia performed worse in verbal-involved PAL (visual-verbal, verbal-visual, and verbal-verbal) than the control children. Experiment 2 revealed that children with dyslexia performed better than the control children in the verbal-visual condition. Our results suggest that children with dyslexia have an intact ability to form cross-modal associations, which also implies that phonological deficits might be the key to PAL deficits in Chinese children with dyslexia.

The Relationship Between Music-Related Types of Synesthesia and Mental Imagery in Synesthete Musicians

Synesthesia is a perceptual and conceptual phenomenon that has been linked to a heightened capacity for mental imagery. However, our understanding of music-related types of synesthesia and mental imagery is still in its infancy. This study therefore aims to examine the relationship between music-related types of synesthesia and mental imagery in a sample of synesthete musicians. Four forms of data collection were employed: an online survey, a synesthesia test battery, an absolute pitch test, and a semi-structured interview. This article focuses on results obtained in response to open-ended questions relating to synesthete musicians’ mental imagery. An analysis of the interview data revealed heightened vividness of auditory imagery (AI), including the ability to control the speed, dynamics, and intensity of the imagery. A bidirectional analysis of the functional relationship between AI and synesthesia ascertained that multimodal mental imagery (MMI) could be triggered by AI, thus bypassing the need for sensory stimulation. Furthermore, cases of complex visual imagery (CVI) were confirmed to meet the synesthetic criteria of being automatic and involuntary, as well as being consistent and stable mappings between specific musical stimuli and CVI. These findings are noteworthy as they call into question the general assumption that synesthetic percepts are generic and simple in nature. Together, these findings contribute to our understanding of music-related types of synesthesia and mental imagery, challenge contemporary understanding of certain categorical aspects of music-related synesthetic percepts, and provide us with a deeper understanding of the nature of cross-modal associations in general.

Implicit Association Test (IAT) Studies Investigating Pitch-Shape Audiovisual Cross-modal Associations Across Language Groups.

Previous studies have shown that Chinese speakers and non-Chinese speakers exhibit different patterns of cross-modal congruence for the lexical tones of Mandarin Chinese, depending on which features of the pitch they attend to. But is this pattern of language-specific listening a conscious cultural strategy or an automatic processing effect? If automatic, does it also apply when the same pitch contours no longer sound like speech? Implicit Association Tests (IATs) provide an indirect measure of cross-modal association. In a series of IAT studies, conducted with participants with three kinds of language backgrounds (Chinese-dominant bilinguals, Chinese balanced bilinguals, and English speakers with no Chinese experience) we find language-specific congruence effects for Mandarin lexical tones but not for matched sine-wave stimuli. That is, for linguistic stimuli, non-Chinese speakers show advantages for pitch-height congruence (high-pointy, low-curvy); no congruence effects were found for Chinese speakers. For non-linguistic stimuli, all participant groups showed advantages for pitch-height congruence. The present findings suggest that non-lexical tone congruence (high-pointy, low-curvy) is a basic congruence pattern, and the acquisition of a language with lexical tone can alter this perception.