Cross-modal Associations Research Articles

CSKINet: A multimodal network model integrating conceptual semantic knowledge injection for relation extraction of Chinese corporate reports

Recognizing the associations among entities in corporate reports accurately is crucial for market regulation and policy development. Nevertheless, confronted with massive corporate information, the traditional manual screening approach is cumbersome, struggling to match the demand. Consequently, we propose a multimodal network model incorporating conceptual semantic knowledge injection, CSKINet, for accurately extracting relations from Chinese corporate reports. The essential highlights in the design of the CSKINet model are the following: (1) Integrate the conceptual descriptions of corporations from external resources to construct the semantic knowledge repository of corporate concepts, which provides a solid semantic foundation for the model. (2) Multimodal features are extracted from the documents by various means and corporate conceptual knowledge is integrated into the model representation to enhance the representation capability of the model. (3) The multimodal self-attention mechanism that captures cross-modal associations and the biaffine classifier with Taylor polynomial loss function that optimizes training iterations further improve the learning efficiency and prediction accuracy. The results on the real corporate report dataset show that our proposed model can more accurately extract the relations from Chinese corporate reports compared to other baseline models, where the F1 score reaches 85.76%.

Fish behavior recognition based on an audio-visual multimodal interactive fusion network

In light of the challenges imposed by fish behavior recognition, which arise from environmental noise and dim lighting in aquaculture environments and adversely affect the effectiveness of unimodal recognition methods based on either sound or visual cues, this paper proposes a fish behavior recognition model, Mul-SEResNet50, based on the fusion of audio and visual information. To address issues such as image blurring and indistinct sounds in aquaculture environments, which hinder the effectiveness of multimodal fusion and complementary modalities, a multimodal interaction fusion (MIF) module is introduced. This module integrates audio-visual modalities at multiple stages to achieve a more comprehensive joint feature representation. To enhance complementarity during the fusion process, we designed a U-shaped bilinear fusion structure to fully utilize multimodal information, capture cross-modal associations, and extract high-level features. Furthermore, to address the potential loss of key features, a temporal aggregation and pooling (TAP) layer is introduced to preserve more fine-grained features by extracting both the maximum and average values within pooling regions. To validate the effectiveness of the proposed model, both ablation experiments and comparative experiments are conducted. The results demonstrate that Mul-SEResNet50 achieves a 5.04 % accuracy improvement over SEResNet50 without sacrificing detection speed. Compared to the state-of-the-art U-FusionNet-ResNet50 +SENet model, Mul-SEResNet50 achieves accuracy and F1 score improvements of 0.47 % and 1.32 %, respectively. These findings confirm the efficacy of the proposed model in terms of accurately recognizing fish behavior, facilitating the precise monitoring of fish behavior.

Variability in odor hedonic perception: A challenge for neurosensory and behavioral research.

Odor hedonic perception (pleasant/unpleasant character) is considered to be the first and one of the most important dimensions in olfaction and is known to be highly variable and dependent on several factors related to the stimulus, individual characteristics, and context. Although numerous experimental studies have been published on this topic, there is no comprehensive general review on the variability in odor hedonic perception. Therefore, the aim of this article was to describe and detail all the factors involved in the variability in odor hedonic perception. Stimulus properties-related variability includes the concentration/intensity and the physicochemical properties. Individual-related variability includes sex, age, and physiological state characteristics, while context-related variability involves factors associated with the stimulation context, such as stimulation pathway, exposure frequency, verbal influences, cross-modal associations, hedonic contrast, and cultural background. Subsequently, a section is devoted to the brain processing of odor hedonicity, although its role in odor hedonic variability is largely unknown. Finally, the article discusses the relationship between the hedonic qualities of odors and the resulting behavioral responses in both animals and humans. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

SARD: Fake news detection based on CLIP contrastive learning and multimodal semantic alignment

The automatic detection of multimodal fake news can be used to effectively identify potential risks in cyberspace. Most of the existing multimodal fake news detection methods focus on fully exploiting textual and visual features in news content, thus neglecting the full utilization of news social context features that play an important role in improving fake news detection. To this end, we propose a new fake news detection method based on CLIP contrastive learning and multimodal semantic alignment (SARD). SARD leverages cutting-edge multimodal learning techniques, such as CLIP, and robust cross-modal contrastive learning methods to integrate features of news-oriented heterogeneous information networks (N-HIN) with multi-level textual and visual features into a unified framework for the first time. This framework not only achieves cross-modal alignment between deep textual and visual features but also considers cross-modal associations and semantic alignments across different modalities. Furthermore, SARD enhances fake news detection by aligning semantic features between news content and N-HIN features, an aspect largely overlooked by existing methods. We test and evaluate SARD on three real-world datasets. Experimental results demonstrate that SARD significantly outperforms the twelve state-of-the-art competitors in fake news detection, with an average improvement of 2.89% in Mac.F1 score and 2.13% in accuracy compared to the leading baseline models across three datasets.

“Spicy Touch”: Cross-modal associations between hand-feel touch and capsaicin-induced oral irritation

The influence of extrinsic hand-feel touch cues on consumer experiences in food and beverage consumption is well established. However, their impact on trigeminal perception, particularly the oral irritation caused by capsaicin or spicy foods, is less understood. This study aimed to determine the existence of cross-modal associations between hand-feel touch and capsaicin-induced oral irritation. This study investigated whether these potential associations were driven by the sensory contributions of the hand-feel tactile materials (measured by instrumental physical parameters) or by affective responses (evaluated through hedonic scales and the self-reported emotion questionnaire, EsSense Profile®, by consumers). In our study, 96 participants tasted a capsaicin solution while engaging with nine hand-feel tactile materials, i.e., cardboard, linen, rattan, silicone, stainless steel, sandpaper (fine), sandpaper (rough), sponge, and towel. They subsequently rated their liking and emotional responses, perceived intensity of oral irritation, and the congruency between hand-feel tactile sensation and oral irritation. Instrumental measurements characterized the surface texture of the hand-feel tactile materials, which were correlated with the collected sensory data. The results revealed that unique cross-modal associations between hand-feel touch and capsaicin-induced oral irritation. Specifically, while sandpapers demonstrated high congruence with the sensation of oral irritation, stainless steel was found to be least congruent. These associations were influenced by both the common emotional responses (“active,” “aggressive,” “daring,” “energetic,” “guilty,” and “worried”) evoked by the hand-feel tactile materials and the capsaicin, as well as by participants’ liking for the hand-feel tactile materials and the characteristics of the surface textures. This study provides empirical evidence of the cross-modality between hand-feel tactile sensations and capsaicin-induced oral irritation, opening new avenues for future research in this area.

Classification of lung cancer subtypes on CT images with synthetic pathological priors

The accurate diagnosis on pathological subtypes for lung cancer is of significant importance for the follow-up treatments and prognosis managements. In this paper, we propose self-generating hybrid feature network (SGHF-Net) for accurately classifying lung cancer subtypes on computed tomography (CT) images. Inspired by studies stating that cross-scale associations exist in the image patterns between the same case’s CT images and its pathological images, we innovatively developed a pathological feature synthetic module (PFSM), which quantitatively maps cross-modality associations through deep neural networks, to derive the “gold standard” information contained in the corresponding pathological images from CT images. Additionally, we designed a radiological feature extraction module (RFEM) to directly acquire CT image information and integrated it with the pathological priors under an effective feature fusion framework, enabling the entire classification model to generate more indicative and specific pathologically related features and eventually output more accurate predictions. The superiority of the proposed model lies in its ability to self-generate hybrid features that contain multi-modality image information based on a single-modality input. To evaluate the effectiveness, adaptability, and generalization ability of our model, we performed extensive experiments on a large-scale multi-center dataset (i.e., 829 cases from three hospitals) to compare our model and a series of state-of-the-art (SOTA) classification models. The experimental results demonstrated the superiority of our model for lung cancer subtypes classification with significant accuracy improvements in terms of accuracy (ACC), area under the curve (AUC), positive predictive value (PPV) and F1-score.

Crossmodal compensation: Sensory discomfort affects consumer preferences across modalities

AbstractConsumers are frequently exposed to uncomfortable sensory circumstances (e.g., hot weather and noise pollution), yet little research has examined how such discomfort might systematically shape preferences. Drawing from research on crossmodal associations, we propose and empirically support a discomfort‐driven crossmodal compensation effect in which consumers exposed to uncomfortable atmospheric sensations in one modality (e.g., loud noise) compensate through product choices across other modalities (e.g., choosing visually “quiet” products). Across five studies, we document the consequences of this phenomenon on actual (studies 1 and 2) and hypothetical (studies 3, 4, and 5) consumer choices and preferences. Further, we demonstrate that a desire to reduce sensory discomfort drives this crossmodal compensation effect, and the spillover only manifests when individuals are unable to adjust within the initially disturbed modality. Our research contributes to the literature on goal theory and crossmodal associations by demonstrating that discomfort experienced in one modality can systematically affect product preferences in other modalities.

Macaque claustrum, pulvinar and putative dorsolateral amygdala support the cross-modal association of social audio-visual stimuli based on meaning.

Social communication draws on several cognitive functions such as perception, emotion recognition and attention. The association of audio-visual information is essential to the processing of species-specific communication signals. In this study, we use functional magnetic resonance imaging in order to identify the subcortical areas involved in the cross-modal association of visual and auditory information based on their common social meaning. We identified three subcortical regions involved in audio-visual processing of species-specific communicative signals: the dorsolateral amygdala, the claustrum and the pulvinar. These regions responded to visual, auditory congruent and audio-visual stimulations. However, none of them was significantly activated when the auditory stimuli were semantically incongruent with the visual context, thus showing an influence of visual context on auditory processing. For example, positive vocalization (coos) activated the three subcortical regions when presented in the context of positive facial expression (lipsmacks) but not when presented in the context of negative facial expression (aggressive faces). In addition, the medial pulvinar and the amygdala presented multisensory integration such that audiovisual stimuli resulted in activations that were significantly higher than those observed for the highest unimodal response. Last, the pulvinar responded in a task-dependent manner, along a specific spatial sensory gradient. We propose that the dorsolateral amygdala, the claustrum and the pulvinar belong to a multisensory network that modulates the perception of visual socioemotional information and vocalizations as a function of the relevance of the stimuli in the social context. SIGNIFICANCE STATEMENT: Understanding and correctly associating socioemotional information across sensory modalities, such that happy faces predict laughter and escape scenes predict screams, is essential when living in complex social groups. With the use of functional magnetic imaging in the awake macaque, we identify three subcortical structures-dorsolateral amygdala, claustrum and pulvinar-that only respond to auditory information that matches the ongoing visual socioemotional context, such as hearing positively valenced coo calls and seeing positively valenced mutual grooming monkeys. We additionally describe task-dependent activations in the pulvinar, organizing along a specific spatial sensory gradient, supporting its role as a network regulator.

Increases in pre-stimulus theta and alpha oscillations precede successful encoding of crossmodal associations

A central aspect of episodic memory is the formation of associations between stimuli from different modalities. Current theoretical approaches assume a functional role of ongoing oscillatory power and phase in the theta band (3–7 Hz) for the encoding of crossmodal associations. Furthermore, ongoing activity in the theta range as well as alpha (8–12 Hz) and low beta activity (13–20 Hz) before the presentation of a stimulus is thought to modulate subsequent cognitive processing, including processes that are related to memory. In this study, we tested the hypothesis that pre-stimulus characteristics of low frequency activity are relevant for the successful formation of crossmodal memory. The experimental design that was used specifically allowed for the investigation of associative memory independent from individual item memory. Participants (n = 51) were required to memorize associations between audiovisual stimulus pairs and distinguish them from newly arranged ones consisting of the same single stimuli in the subsequent recognition task. Our results show significant differences in the state of pre-stimulus theta and alpha power between remembered and not remembered crossmodal associations, clearly relating increased power to successful recognition. These differences were positively correlated with memory performance, suggesting functional relevance for behavioral measures of associative memory. Further analysis revealed similar effects in the low beta frequency ranges, indicating the involvement of different pre-stimulus-related cognitive processes. Phase-based connectivity measures in the theta band did not differ between remembered and not remembered stimulus pairs. The findings support the assumed functional relevance of theta band oscillations for the formation of associative memory and demonstrate that an increase of theta as well as alpha band oscillations in the pre-stimulus period is beneficial for the establishment of crossmodal memory.

Improving Cross-Modal Alignment with Synthetic Pairs for Text-Only Image Captioning

Although image captioning models have made significant advancements in recent years, the majority of them heavily depend on high-quality datasets containing paired images and texts which are costly to acquire. Previous works leverage the CLIP's cross-modal association ability for image captioning, relying solely on textual information under unsupervised settings. However, not only does a modality gap exist between CLIP text and image features, but a discrepancy also arises between training and inference due to the unavailability of real-world images, which hinders the cross-modal alignment in text-only captioning. This paper proposes a novel method to address these issues by incorporating synthetic image-text pairs. A pre-trained text-to-image model is deployed to obtain images that correspond to textual data, and the pseudo features of generated images are optimized toward the real ones in the CLIP embedding space. Furthermore, textual information is gathered to represent image features, resulting in the image features with various semantics and the bridged modality gap. To unify training and inference, synthetic image features would serve as the training prefix for the language decoder, while real images are used for inference. Additionally, salient objects in images are detected as assistance to enhance the learning of modality alignment. Experimental results demonstrate that our method obtains the state-of-the-art performance on benchmark datasets.

Crossmodal associations between naturally occurring tactile and sound textures.

This study investigates the crossmodal associations between naturally occurring sound textures and tactile textures. Previous research has demonstrated the association between low-level sensory features of sound and touch, as well as higher-level, cognitively mediated associations involving language, emotions, and metaphors. However, stimuli like textures, which are found in both modalities have received less attention. In this study, we conducted two experiments: a free association task and a two alternate forced choice task using everyday tactile textures and sound textures selected from natural sound categories. The results revealed consistent crossmodal associations reported by participants between the textures of the two modalities. They tended to associate more sound textures (e.g., wood shavings and sandpaper) with tactile surfaces that were rated as harder, rougher, and intermediate on the sticky-slippery scale. While some participants based the auditory-tactile association on sensory features, others made the associations based on semantic relationships, co-occurrence in nature, and emotional mediation. Interestingly, the statistical features of the sound textures (mean, variance, kurtosis, power, autocorrelation, and correlation) did not show significant correlations with the crossmodal associations, indicating a higher-level association. This study provides insights into auditory-tactile associations by highlighting the role of sensory and emotional (or cognitive) factors in prompting these associations.

Grounded language acquisition through the eyes and ears of a single child.

Starting around 6 to 9 months of age, children begin acquiring their first words, linking spoken words to their visual counterparts. How much of this knowledge is learnable from sensory input with relatively generic learning mechanisms, and how much requires stronger inductive biases? Using longitudinal head-mounted camera recordings from one child aged 6 to 25 months, we trained a relatively generic neural network on 61 hours of correlated visual-linguistic data streams, learning feature-based representations and cross-modal associations. Our model acquires many word-referent mappings present in the child's everyday experience, enables zero-shot generalization to new visual referents, and aligns its visual and linguistic conceptual systems. These results show how critical aspects of grounded word meaning are learnable through joint representation and associative learning from one child's input.

Prokofiev was (almost) right: A cross-cultural investigation of auditory-conceptual associations in Peter and the Wolf

Over recent decades, studies investigating cross-modal correspondences have documented the existence of a wide range of consistent cross-modal associations between simple auditory and visual stimuli or dimensions (e.g., pitch-lightness). Far fewer studies have investigated the association between complex and realistic auditory stimuli and visually presented concepts (e.g., musical excerpts-animals). Surprisingly, however, there is little evidence concerning the extent to which these associations are shared across cultures. To address this gap in the literature, two experiments using a set of stimuli based on Prokofiev’s symphonic fairy tale Peter and the Wolf are reported. In Experiment 1, 293 participants from several countries and with very different language backgrounds rated the association between the musical excerpts, images and words representing the story’s characters (namely, bird, duck, wolf, cat, and grandfather). The results revealed that participants tended to consistently associate the wolf and the bird with the corresponding musical excerpt, while the stimuli of other characters were not consistently matched across participants. Remarkably, neither the participants’ cultural background, nor their musical expertise affected the ratings. In Experiment 2, 104 participants were invited to rate each stimulus on eight emotional features. The results revealed that the emotional profiles associated with the music and with the concept of the wolf and the bird were perceived as more consistent between observers than the emotional profiles associated with the music and the concept of the duck, the cat, and the grandpa. Taken together, these findings therefore suggest that certain auditory-conceptual associations are perceived consistently across cultures and may be mediated by emotional associations.

Crossmodal Associations and Working Memory in the Brain.

Crossmodal associations between stimuli from different sensory modalities could emerge in non-synesthetic people and be stored in working memory to guide goal-directed behaviors. This chapter reviews a plethora of studies in this field to summarize where, when, and how crossmodal associations and working memory are processed. It has been found that in those brain regions that are traditionally considered as unimodal primary sensory areas, neural activity could be influenced by crossmodal sensory signals at temporally very early stage of information processing. This phenomenon could not be due to feedback projections from higher level associative areas. Sequentially, neural processes would then occur in associative cortical areas including the posterior parietal cortex and prefrontal cortex. Neural oscillations in multiple frequency bands may reflect brain activity in crossmodal associations, and it is likely that neural synchrony is related to potential neural mechanisms underlying these processes. Primary sensory areas and associative areas coordinate together through neural synchrony to fulfil crossmodal associations and to guide working memory performance.

MCD-Net: Toward RGB-D Video Inpainting in Real-World Scenes.

Video inpainting gains an increasing amount of attention ascribed to its wide applications in intelligent video editing. However, despite tremendous progress made in RGB video inpainting, the existing RGB-D video inpainting models are still incompetent to inpaint real-world RGB-D videos, as they simply fuse color and depth via explicit feature concatenation, neglecting the natural modality gap. Moreover, current RGB-D video inpainting datasets are synthesized with homogeneous and delusive RGB-D data, which is far from real-world application and cannot provide comprehensive evaluation. To alleviate these problems and achieve real-world RGB-D video inpainting, on one hand, we propose a Mutually-guided Color and Depth Inpainting Network (MCD-Net), where color and depth are reciprocally leveraged to inpaint each other implicitly, mitigating the modality gap and fully exploiting cross-modal association for inpainting. On the other hand, we build a Video Inpainting with Depth (VID) dataset to supply diverse and authentic RGB-D video data with various object annotation masks to enable comprehensive evaluation for RGB-D video inpainting under real-world scenes. Experimental results on the DynaFill benchmark and our collected VID dataset demonstrate our MCD-Net not only yields the state-of-the-art quantitative performance but successfully achieves high-quality RGB-D video inpainting under real-world scenes. All resources are available at https://github.com/JCATCV/MCD-Net.

Image Understands Point Cloud: Weakly Supervised 3D Semantic Segmentation via Association Learning.

Weakly supervised point cloud semantic segmentation methods that require 1% or fewer labels with the aim of realizing almost the same performance as fully supervised approaches have recently attracted extensive research attention. A typical solution in this framework is to use self-training or pseudo-labeling to mine the supervision from the point cloud itself while ignoring the critical information from images. In fact, cameras widely exist in LiDAR scenarios, and this complementary information seems to be highly important for 3D applications. In this paper, we propose a novel cross-modality weakly supervised method for 3D segmentation that incorporates complementary information from unlabeled images. We design a dual-branch network equipped with an active labeling strategy to maximize the power of tiny parts of labels and to directly realize 2D-to-3D knowledge transfer. Afterward, we establish a cross-modal self-training framework, which iterates between parameter updating and pseudolabel estimation. In the training phase, we propose cross-modal association learning to mine complementary supervision from images by reinforcing the cycle consistency between 3D points and 2D superpixels. In the pseudolabel estimation phase, a pseudolabel self-rectification mechanism is derived to filter noisy labels, thus providing more accurate labels for the networks to be fully trained. The extensive experimental results demonstrate that our method even outperforms the state-of-the-art fully supervised competitors with less than 1% actively selected annotations.

現代漢語和古漢語的通感隱喩映射方向性綜合硏究

Synaesthesia is a physical phenomenon which refers to ‘the involuntary experience of a cross-modal association’ (Shen 2008). In linguistics, generally speaking, synaesthesia is considered as a common sort of metaphor in poetic and ordinary language which can make a metaphoric transfer from one sensory domain to another. For instance, ‘cold color’ is linguistically synaesthetic, because the speaker expresses a perception of vision by using a word related to touch. In this paper, I make a brief introduction to linguistic synaesthesia, namely, synaesthetic metaphor, based on previous studies in different languages and different methods, and then, attempt to analyze the synaesthetic data collected from modern and ancient Chinese. In conclusion, the data analysis results of both modern Mandarin Chinese and ancient Chinese show that the linguistic synaesthesia from Chinese is different from that from Ullmann’s (1957) study in the metaphoric directionality. It implies that linguistic synaesthesia could have language- or culture-specific variations as per language.

Cross-Modal Contributions to Episodic Memory for Voices.

Multisensory context often facilitates perception and memory. In fact, encoding items within a multisensory context can improve memory even on strictly unisensory tests (i.e., when the multisensory context is absent). Prior studies that have consistently found these multisensory facilitation effects have largely employed multisensory contexts in which the stimuli were meaningfully related to the items targeting for remembering (e.g., pairing canonical sounds and images). Other studies have used unrelated stimuli as multisensory context. A third possible type of multisensory context is one that is environmentally related simply because the stimuli are often encountered together in the real world. We predicted that encountering such a multisensory context would also enhance memory through cross-modal associations, or representations relating to one's prior multisensory experience with that sort of stimuli in general. In two memory experiments, we used faces and voices of unfamiliar people as everyday stimuli individuals have substantial experience integrating the perceptual features of. We assigned participants to face- or voice-recognition groups and ensured that, during the study phase, half of the face or voice targets were encountered also with information in the other modality. Voices initially encoded along with faces were consistently remembered better, providing evidence that cross-modal associations could explain the observed multisensory facilitation.

Children’s multisensory experiences in museums: how olfaction interacts with color

This case study was designed to engage children’s sense of smell through a story-related museum exhibition. Children’s responses to the exhibition, with particular attention to their olfactory perceptions of the odors at the exhibition, were solicited through researcher-child interviews and children’s drawings. Responses from 28 children (girls N = 14, boys N = 14) aged between 4.5–8 years were analyzed after they visited the exhibition using the cross-modal association and multisensory theories. Interview data showed that dark (brown and black) colors elicited children’s negative olfactory associations for both positive and negative odors. Children’s drawings did not seem to make references to the odors at the exhibitions but rather their preferences for the different story characters. We theorize about the associations between smell and colors in children’s responses and distil some key learnings for multisensory museology.

Memristive circuit design and implementation of olfactory perception and cross-modal association

Memristive circuit design and implementation of olfactory perception and cross-modal association