Intermodal Interactions Research Articles

A 3D hierarchical cross-modality interaction network using transformers and convolutions for brain glioma segmentation in MR images.

Precise glioma segmentation from multi-parametric magnetic resonance (MR) images is essential for brain glioma diagnosis. However, due to the indistinct boundaries between tumor sub-regions and the heterogeneous appearances of gliomas in volumetric MR scans, designing a reliable and automated glioma segmentation method is still challenging. Although existing 3D Transformer-based or convolution-based segmentation networks have obtained promising results via multi-modal feature fusion strategies or contextual learning methods, they widely lack the capability of hierarchical interactions between different modalities and cannot effectively learn comprehensive feature representations related to all glioma sub-regions. To overcome these problems, in this paper, we propose a 3D hierarchical cross-modality interaction network (HCMINet) using Transformers and convolutions for accurate multi-modal glioma segmentation, which leverages an effective hierarchical cross-modality interaction strategy to sufficiently learn modality-specific and modality-shared knowledge correlated to glioma sub-region segmentation from multi-parametric MR images. In the HCMINet, we first design a hierarchical cross-modality interaction Transformer (HCMITrans) encoder to hierarchically encode and fuse heterogeneous multi-modal features by Transformer-based intra-modal embeddings and inter-modal interactions in multiple encoding stages, which effectively captures complex cross-modality correlations while modeling global contexts. Then, we collaborate an HCMITrans encoder with a modality-shared convolutional encoder to construct the dual-encoder architecture in the encoding stage, which can learn the abundant contextual information from global and local perspectives. Finally, in the decoding stage, we present a progressive hybrid context fusion (PHCF) decoder to progressively fuse local and global features extracted by the dual-encoder architecture, which utilizes the local-global context fusion (LGCF) module to efficiently alleviate the contextual discrepancy among the decoding features. Extensive experiments are conducted on two public and competitive glioma benchmark datasets, including the BraTS2020 dataset with 494 patients and the BraTS2021 dataset with 1251 patients. Results show that our proposed method outperforms existing Transformer-based and CNN-based methods using other multi-modal fusion strategies in our experiments. Specifically, the proposed HCMINet achieves state-of-the-art mean DSC values of 85.33% and 91.09% on the BraTS2020 online validation dataset and the BraTS2021 local testing dataset, respectively. Our proposed method can accurately and automatically segment glioma regions from multi-parametric MR images, which is beneficial for the quantitative analysis of brain gliomas and helpful for reducing the annotation burden of neuroradiologists.

Prevention and adaptation of intermodal interactive seaports and dry ports under asymmetric risk behavior

The port-hinterland system, encompassing seaports and dry ports, has evolved into the backbone of international trade. Due to their unique geographic locations and critical socioeconomic functions, ports are particularly susceptible to a range of risks. Port authorities (PAs) can address these challenges through targeted preventive efforts, which aims to reduce the probability of risk occurrence, and adaptive efforts, which seeks to mitigate potential damage. However, intermodal interactions and the asymmetric risk behaviors exhibited by ports under risk probability ambiguity present significant challenges for PAs making risk-responsive investments. This study develops a two-stage game model to examine the effects of asymmetric risk behavior and risk probability ambiguity on strategic investment decisions. Our analysis reveals that prevention are susceptible to free-rider effects, which are further exacerbated by risk behaviors and mitigated by investment strategy differentiation. Moreover, we demonstrate that, as a general guideline, seaport authorities should prioritize adaptation, while dry port authorities should focus on prevention, with adaptation being the primary strategy when the level of risk aversion is low. These findings provide valuable managerial insights for improving the risk management and resilience of the port-hinterland system.

Spatial-Temporal Co-Attention Learning for Diagnosis of Mental Disorders From Resting-State fMRI Data.

Neuroimaging techniques have been widely adopted to detect the neurological brain structures and functions of the nervous system. As an effective noninvasive neuroimaging technique, functional magnetic resonance imaging (fMRI) has been extensively used in computer-aided diagnosis (CAD) of mental disorders, e.g., autism spectrum disorder (ASD) and attention deficit/hyperactivity disorder (ADHD). In this study, we propose a spatial-temporal co-attention learning (STCAL) model for diagnosing ASD and ADHD from fMRI data. In particular, a guided co-attention (GCA) module is developed to model the intermodal interactions of spatial and temporal signal patterns. A novel sliding cluster attention module is designed to address global feature dependency of self-attention mechanism in fMRI time series. Comprehensive experimental results demonstrate that our STCAL model can achieve competitive accuracies of 73.0 ± 4.5%, 72.0 ± 3.8%, and 72.5 ± 4.2% on the ABIDE I, ABIDE II, and ADHD-200 datasets, respectively. Moreover, the potential for feature pruning based on the co-attention scores is validated by the simulation experiment. The clinical interpretation analysis of STCAL can allow medical professionals to concentrate on the discriminative regions of interest and key time frames from fMRI data.

Joint training strategy of unimodal and multimodal for multimodal sentiment analysis

With the explosive growth of social media video content, research on multimodal sentiment analysis (MSA) has attracted considerable attention recently. Despite significant progress in MSA, there remains challenges: current research mostly focuses on learning either unimodal features or aspects of multimodal interactions, neglecting the importance of simultaneously considering both unimodal features and intermodal interactions. To address the aforementioned challenges, this paper proposes a fusion strategy called Joint Training of Unimodal and Multimodal (JTUM). Specifically, this strategy combines unimodal label generation module with cross-modal transformer. The unimodal label generation module aims to generate more distinctive labels for each unimodal input, facilitating more effective learning of unimodal representations. Meanwhile, cross-modal transformer is designed to treat each modality as a target modality and optimize it using other modalities as source modalities, thereby learning the interactions between each pair of modalities. By jointly training unimodal and multimodal tasks, our model can focus on individual modality features while learning the interactions between modalities. Finally, to better capture temporal information and make predictions, we also added self-attention transformer as sequence models. Experimental results on the CMU-MOSI and CMU-MOSEI datasets demonstrate that JTUM outperforms current main methods.

Follower attraction in live streaming: Knowledge driven by PKM and data driven by EF-LSTM

In this study, we examine variables affecting follower attraction and determine the optimal model for its prediction. We identify 12 variables from the persuasion attempt perspective in the persuasion knowledge model and accordingly perform two progressive studies (study 1: knowledge driven, with coarse-grained variables, with unimodal models, without data fusion; study 2: knowledge-and-data driven, with fine-grained features, with multimodal models, with data fusion). In study 1, we compute 12 variables in accordance with hypotheses from nonfused trimodal data (verbal, vocal, and visual). XGBoost and Shapley additive explanations are used to evaluate the importance of these 12 variables, and economic models are adopted to test their statistical significance. In study 2, we reduce the data granularity of the 12 coarse-grained variables by generating 323 fine-grained features. We then process trimodal data fusion to consider intra- and intermodal interactions. Long Short-Term Memory (LSTM) models are applied to learn a mapping from the input sequences of trimodality to output values of follower increment. The results demonstrate that early fusion LSTM with fused trimodal features has a 66.3% prediction accuracy, outperforming LSTM with unimodal features and late fusion LSTM with nonfused trimodal features. To our knowledge, these results are among the first to unfold follower attraction strategies with unstructured video data, and this is the first study to provide knowledge-and-data-driven multimodal machine learning models for predicting follower attraction at different granularity levels.

Hierarchical multimodal self-attention-based graph neural network for DTI prediction.

Drug-target interactions (DTIs) are a key part of drug development process and their accurate and efficient prediction can significantly boost development efficiency and reduce development time. Recent years have witnessed the rapid advancement of deep learning, resulting in an abundance of deep learning-based models for DTI prediction. However, most of these models used a single representation of drugs and proteins, making it difficult to comprehensively represent their characteristics. Multimodal data fusion can effectively compensate for the limitations of single-modal data. However, existing multimodal models for DTI prediction do not take into account both intra- and inter-modal interactions simultaneously, resulting in limited presentation capabilities of fused features and a reduction in DTI prediction accuracy. A hierarchical multimodal self-attention-based graph neural network for DTI prediction, called HMSA-DTI, is proposed to address multimodal feature fusion. Our proposed HMSA-DTI takes drug SMILES, drug molecular graphs, protein sequences and protein 2-mer sequences as inputs, and utilizes a hierarchical multimodal self-attention mechanism to achieve deep fusion of multimodal features of drugs and proteins, enabling the capture of intra- and inter-modal interactions between drugs and proteins. It is demonstrated that our proposed HMSA-DTI has significant advantages over other baseline methods on multiple evaluation metrics across five benchmark datasets.

A Multimodal Sentiment Analysis Approach Based on a Joint Chained Interactive Attention Mechanism

The objective of multimodal sentiment analysis is to extract and integrate feature information from text, image, and audio data accurately, in order to identify the emotional state of the speaker. While multimodal fusion schemes have made some progress in this research field, previous studies still lack adequate approaches for handling inter-modal information consistency and the fusion of different categorical features within a single modality. This study aims to effectively extract sentiment coherence information among video, audio, and text and consequently proposes a multimodal sentiment analysis method named joint chain interactive attention (VAE-JCIA, Video Audio Essay–Joint Chain Interactive Attention). In this approach, a 3D CNN is employed for extracting facial features from video, a Conformer is employed for extracting audio features, and a Funnel-Transformer is employed for extracting text features. Furthermore, the joint attention mechanism is utilized to identify key regions where sentiment information remains consistent across video, audio, and text. This process acquires reinforcing features that encapsulate information regarding consistency among the other two modalities. Inter-modal feature interactions are addressed through chained interactive attention, and multimodal feature fusion is employed to efficiently perform emotion classification. The method is experimentally validated on the CMU-MOSEI dataset and the IEMOCAP dataset. The experimental results demonstrate that the proposed method significantly enhances the performance of the multimodal sentiment analysis model.

Emotion-aware hierarchical interaction network for multimodal image aesthetics assessment

Image aesthetics assessment (IAA) has attracted increasing attention recently but is still challenging due to its high abstraction and complexity. Intuitively, image emotion and aesthetics are both human subjective feelings evoked by visual content. Previous researches have demonstrated that image aesthetics has intrinsic relationships with emotion. In fact, human emotional experience has potential impact on the perception of image aesthetics. Therefore, emotion information can be exploited to enhance aesthetic representation learning. Inspired by this, this paper presents an Emotion-Aware Hierarchical Interaction NETwork (EAHI-NET) for multimodal image aesthetics assessment, which explores both intra-modal and inter-modal interactions between aesthetics and emotions hierarchically. Specifically, we first propose the intra-modal emotion-aesthetics interaction module to obtain emotion-enhanced visual and textual aesthetic representations respectively. Then we propose the inter-modal feature enhancement to obtain the cross-modal aesthetic and emotion features. Finally, we design the inter-modal emotion-aesthetics interaction module to further investigate the cross-modal interplay between aesthetics and emotion, based on which hierarchical feature representations are achieved for multimodal IAA. The extensive experiments show that the proposed method can outperform the state-of-the-arts on multimodal AVA and Photo.net datasets.

Nonlinear scaling of fluctuation kinetic energy for shock–vorticity wave interaction

Turbulent kinetic energy (TKE) is a quantity of primary importance in shock–turbulence interaction (STI). Linear interaction analysis (LIA) serves as a theoretical method to predict the amplification of TKE in STI. LIA is constrained by low-amplitude fluctuations and neglects nonlinear effects. In this paper, we explore the nonlinear amplification of the kinetic energy of fluctuations in the interaction between a vorticity wave and a shock wave that serves as a building block flow for STI. A weakly nonlinear framework (WNLF) is introduced to analyze nonlinear effects in fluctuation kinetic energy (FKE) and identify the dominant physical mechanisms driving its amplification. The theoretical framework is validated through high-accuracy numerical simulations of shock–vorticity wave interactions. The simulation results are compared with the predictions derived from the WNLF for a range of intensities and inclinations of shock-upstream vorticity fluctuations at different Mach numbers, and WNLF is found to successfully scale the numerical data for FKE, thus confirming the validity and applicability of the framework. According to WNLF findings, at lower supersonic Mach numbers, the intermodal interaction between vorticity–vorticity modes is important, whereas the interaction between vorticity and acoustic modes becomes dominant at higher Mach numbers. Using the intermodal interactions, a model based on WNLF is proposed to predict TKE amplification in STI. In comparison to direct numerical simulation, the WNLF based model predicts the TKE amplification for moderate turbulent Mach number and lower supersonic flow Mach numbers. This is a significant improvement over the LIA results available in the literature.

A Dual-Attention Learning Network With Word and Sentence Embedding for Medical Visual Question Answering.

Research in medical visual question answering (MVQA) can contribute to the development of computer-aided diagnosis. MVQA is a task that aims to predict accurate and convincing answers based on given medical images and associated natural language questions. This task requires extracting medical knowledge-rich feature content and making fine-grained understandings of them. Therefore, constructing an effective feature extraction and understanding scheme are keys to modeling. Existing MVQA question extraction schemes mainly focus on word information, ignoring medical information in the text, such as medical concepts and domain-specific terms. Meanwhile, some visual and textual feature understanding schemes cannot effectively capture the correlation between regions and keywords for reasonable visual reasoning. In this study, a dual-attention learning network with word and sentence embedding (DALNet-WSE) is proposed. We design a module, transformer with sentence embedding (TSE), to extract a double embedding representation of questions containing keywords and medical information. A dual-attention learning (DAL) module consisting of self-attention and guided attention is proposed to model intensive intramodal and intermodal interactions. With multiple DAL modules (DALs), learning visual and textual co-attention can increase the granularity of understanding and improve visual reasoning. Experimental results on the ImageCLEF 2019 VQA-MED (VQA-MED 2019) and VQA-RAD datasets demonstrate that our proposed method outperforms previous state-of-the-art methods. According to the ablation studies and Grad-CAM maps, DALNet-WSE can extract rich textual information and has strong visual reasoning ability.

QueryTrack: Joint-Modality Query Fusion Network for RGBT Tracking.

Existing RGB-Thermal trackers usually treat intra-modal feature extraction and inter-modal feature fusion as two separate processes, therefore the mutual promotion of extraction and fusion is neglected. Then, the complementary advantages of RGB-T fusion are not fully exploited, and the independent feature extraction is not adaptive to modal quality fluctuation during tracking. To address the limitations, we design a joint-modality query fusion network, in which the intra-modal feature extraction and the inter-modal fusion are coupled together and promote each other via joint-modality queries. The queries are initialized based on the multimodal features of the current frame, making the subsequent fusion adaptive to modal quality fluctuation during tracking. Then the joint-modality query fusion (JQF) utilizes the queries to interact with RGB-T features, allowing the intra-modal enhancement and the inter-modal interactions to be unified for mutual promotion. In this way, JQF can distinguish and enhance the complementary modality features, while filtering out redundant information. For real-time tracking, we propose regional cross-attention for cross-modal interactions to reduce computational cost. Our end-to-end tracker sets a new state-of-the-art performance on multiple RGBT tracking benchmarks including LasHeR, VTUAV, RGBT234 and GTOT, while running at a real-time speed.

Hybrid cross-modal interaction learning for multimodal sentiment analysis

Multimodal sentiment analysis (MSA) predicts the sentiment polarity of an unlabeled utterance that carries multiple modalities, such as text, vision and audio, by analyzing labeled utterances. Existing fusion methods mainly focus on establishing the relationship of characteristics among different modalities to enhance their emotion recognition abilities. However, they always ignore the all-round interaction between different modalities, especially the cross-modal interaction, which is critical to the sentiment decision of multimodal data. To address these issues, we propose a novel hybrid cross-modal interaction learning (HCIL) framework for hybrid learning of intra-modal, inter-modal, interactive-modal and cross-modal interactions, with which the model can fully utilize the sentiment information of multimodalities and enhance the sentiment assistance between modalities. Specifically, we propose two core substructures to learn discriminative multimodal features. One is the comparative learning interaction structure that can track the class dynamics in the intra-modal, reduce the modal gap in the inter-modal and establish emotional communication in the interactive-modal; the other is the cross-modal prediction structure, which can build the sentiment relationship between cross-modal pairs, especially exploring the auxiliary sentiment effect of audio on the vision and text. Furthermore, we adopt a hierarchical feature fusion structure to generate the multimodal feature for the final sentiment prediction. Extensive experiments on three benchmark datasets showed that our HCIL approach can obtain significant performance on the MSA task and that the design of a cross-modal interaction structure can directly promote the improvement of sentiment classification performance.

Hierarchical Synergy-Enhanced Multimodal Relational Network for Video Question Answering

Video question answering (VideoQA) is challenging as it requires reasoning about natural language and multimodal interactive relations. Most existing methods apply attention mechanisms to extract interactions between the question and the video or to extract effective spatio-temporal relational representations. However, these methods neglect the implication of relations between intra- and inter-modal interactions for multimodal learning, and they fail to fully exploit the synergistic effect of multiscale semantics in answer reasoning. In this article, we propose a novel hierarchical synergy-enhanced multimodal relational network (HMRNet) to address these issues. Specifically, we devise (i) a compact and unified relation-oriented interaction module that explores the relation between intra- and inter-modal interactions to enable effective multimodal learning; and (ii) a hierarchical synergistic memory unit that leverages a memory-based interaction scheme to complement and fuse multimodal semantics at multiple scales to achieve synergistic enhancement of answer reasoning. With careful design of each component, our HMRNet has fewer parameters and is computationally efficient. Extensive experiments and qualitative analyses demonstrate that the HMRNet is superior to previous state-of-the-art methods on eight benchmark datasets. We also demonstrate the effectiveness of the different components of our method.

An attention-based multi-modal MRI fusion model for major depressive disorder diagnosis

Objective. Major depressive disorder (MDD) is one of the biggest threats to human mental health. MDD is characterized by aberrant changes in both structure and function of the brain. Although recent studies have developed some deep learning models based on multi-modal magnetic resonance imaging (MRI) for MDD diagnosis, the latent associations between deep features derived from different modalities were largely unexplored by previous studies, which we hypothesized may have potential benefits in improving the diagnostic accuracy of MDD. Approach. In this study, we proposed a novel deep learning model that fused both structural MRI (sMRI) and resting-state MRI (rs-fMRI) data to enhance the diagnosis of MDD by capturing the interactions between deep features extracted from different modalities. Specifically, we first employed a brain function encoder (BFE) and a brain structure encoder (BSE) to extract the deep features from fMRI and sMRI, respectively. Then, we designed a function and structure co-attention fusion (FSCF) module that captured inter-modal interactions and adaptively fused multi-modal deep features for MDD diagnosis. Main results. This model was evaluated on a large cohort and achieved a high classification accuracy of 75.2% for MDD diagnosis. Moreover, the attention distribution of the FSCF module assigned higher attention weights to structural features than functional features for diagnosing MDD. Significance. The high classification accuracy highlights the effectiveness and potential clinical of the proposed model.

A co-attention based multi-modal fusion network for review helpfulness prediction

The current review helpfulness prediction (RHP) methods simply rely on the textual features and meta features to predict review helpfulness, overlooking the informational value of images. Besides, hand-crafted and deep features of text and images have unique advantages, but the combination of them is rarely considered in previous studies. To address these issues, this paper proposes a novel end-to-end architecture utilizing hand-crafted and deep features of text and images simultaneously for RHP. First, the self-attention mechanism considers the intra-modal correlation between hand-crafted and deep features by weighting features at all positions of text and images. Second, a co-attention mechanism is designed to explore dependencies between text and image modality. Third, multi-modalities are fused by simultaneously considering intra-modal and inter-modal interactions for helpfulness prediction. Our proposed framework is verified by two real-world datasets collected from Yelp.com and Amazon.com respectively. The experimental results confirm the favorable performance of our model compared with the benchmark methods. The findings of this study are expected to raise attention to images laden in online reviews, and the complementarity between texts and images from scholars and practitioners.

Intangible cultural heritage image classification with multimodal attention and hierarchical fusion

Designing an efficient Intangible Cultural Heritage (ICH) image classification is beneficial for the public to recognize the ICH and fostering the preservation and spread of that. Currently, related ICH image classification researches mainly focus on the visual features of ICH images, ignoring attached textual descriptions. However, attached textual descriptions can provide crucial clues for ICH images classification. Therefore, in this study, we propose to combine attached textual descriptions to perform ICH image classification in a multimodal way. Additionally, to capture intra- and inter-interactions between ICH images and attached textual descriptions, we propose a novel model named MICMLF, mainly consisted of multimodal attention and hierarchical fusion. Multimodal attention is employed to make the model focus on “important regions” and “important words” in ICH image and attached textual descriptions respectively. Hierarchical fusion is utilized to capture inter-modal dynamics interactions. Extensive experiments are conducted on two Chinese nation-level ICH lists datasets, New Year Print (年画) and Clay Figurine (泥塑). Experimental results demonstrate the superiority of MICMLF, compared with several state-of-the-art methods. Also, the proposed model can handle the situation where ICH images and textual descriptions are incomplete. To the best of our knowledge, we are the first to propose to combine textual descriptions to perform ICH image classification in a multimodal way.

Multi-modal spatial relational attention networks for visual question answering

Visual Question Answering (VQA) is a task that requires VQA model to fully understand the visual information of the image and the language information of the question, and then combine both to provide an answer. Recently, a large amount of VQA approaches focus on modeling intra- and inter-modal interactions with respect to vision and language using a deep modular co-attention network, which can achieve a good performance. Despite their benefits, they also have their limitations. First, the question representation is obtained through Glove word embeddings and Recurrent Neural Network, which may not be sufficient to capture the intricate semantics of the question features. Second, they mostly use visual appearance features extracted by Faster R-CNN to interact with language features, and they ignore important spatial relations between objects in images, resulting in incomplete use of image information. To overcome the limitations of previous methods, we propose a novel Multi-modal Spatial Relation Attention Network (MSRAN) for VQA, which can introduce spatial relationships between objects to fully utilize the image information, thus improving the performance of VQA. In order to achieve the above, we design two types of spatial relational attention modules to comprehensively explore the attention schemes: (i) Self-Attention based on Explicit Spatial Relation (SA-ESR) module that explores geometric relationships between objects explicitly; and (ii) Self-Attention based on Implicit Spatial Relation (SA-ISR) module that can capture the hidden dynamic relationships between objects by using spatial relationship. Moreover, the pre-training model BERT, which replaces Glove word embeddings and Recurrent Neural Network, is applied to MSRAN in order to obtain the better question representation. Extensive experiments on two large benchmark datasets, VQA 2.0 and GQA, demonstrate that our proposed model achieves the state-of-the-art performance.

Multimodal Sentiment Analysis Based on Attentional Temporal Convolutional Network and Multi-Layer Feature Fusion

Multimodal sentiment analysis aims to extract and integrate information from different modalities to accurately identify the sentiment expressed in multimodal data. How to effectively capture the relevant information within a specific modality and how to fully exploit the complementary information among multiple modalities are two major challenges in multimodal sentiment analysis. Traditional approaches fail to obtain the global contextual information of long time-series data when extracting unimodal temporal features, and they usually fuse the features from multiple modalities with the same method and ignore the correlation between different modalities when modeling inter-modal interactions. In this paper, we first propose an Attentional Temporal Convolutional Network (ATCN) to extract unimodal temporal features for enhancing the feature representation ability, then introduce a Multi-layer Feature Fusion (MFF) model to improve the effectiveness of multimodal fusion, which fuses the different-level features by different methods according to the correlation coefficient between the features, and cross-modal multi-head attention is used to fully explore the potential relationship between the low-level features. The experimental results on SIMS and CMU-MOSI datasets show that the proposed model achieves superior performance on sentiment analysis tasks compared to state-of-the-art baselines.

Transformer-Based Multi-Modal Data Fusion Method for COPD Classification and Physiological and Biochemical Indicators Identification.

As the number of modalities in biomedical data continues to increase, the significance of multi-modal data becomes evident in capturing complex relationships between biological processes, thereby complementing disease classification. However, the current multi-modal fusion methods for biomedical data require more effective exploitation of intra- and inter-modal interactions, and the application of powerful fusion methods to biomedical data is relatively rare. In this paper, we propose a novel multi-modal data fusion method that addresses these limitations. Our proposed method utilizes a graph neural network and a 3D convolutional network to identify intra-modal relationships. By doing so, we can extract meaningful features from each modality, preserving crucial information. To fuse information from different modalities, we employ the Low-rank Multi-modal Fusion method, which effectively integrates multiple modalities while reducing noise and redundancy. Additionally, our method incorporates the Cross-modal Transformer to automatically learn relationships between different modalities, facilitating enhanced information exchange and representation. We validate the effectiveness of our proposed method using lung CT imaging data and physiological and biochemical data obtained from patients diagnosed with Chronic Obstructive Pulmonary Disease (COPD). Our method demonstrates superior performance compared to various fusion methods and their variants in terms of disease classification accuracy.

A Visually Enhanced Neural Encoder for Synset Induction

The synset induction task is to automatically cluster semantically identical instances, which are often represented by texts and images. Previous works mainly consider textual parts, while ignoring the visual counterparts. However, how to effectively employ the visual information to enhance the semantic representation for the synset induction is challenging. In this paper, we propose a Visually Enhanced NeUral Encoder (i.e., VENUE) to learn a multimodal representation for the synset induction task. The key insight lies in how to construct multimodal representations through intra-modal and inter-modal interactions among images and text. Specifically, we first design the visual interaction module through the attention mechanism to capture the correlation among images. To obtain the multi-granularity textual representations, we fuse the pre-trained tags and word embeddings. Second, we design a masking module to filter out weakly relevant visual information. Third, we present a gating module to adaptively regulate the modalities’ contributions to semantics. A triplet loss is adopted to train the VENUE encoder for learning discriminative multimodal representations. Then, we perform clustering algorithms on the obtained representations to induce synsets. To verify our approach, we collect a multimodal dataset, i.e., MMAI-Synset, and conduct extensive experiments. The experimental results demonstrate that our method outperforms strong baselines on three groups of evaluation metrics.