Intermodal Interaction Research Articles

Image-text crisis tweet categorization:a caption-based approach

The growth of social media usage this last decade has made available a massive and valuable volume of multimedia data. However, the lack of large multimodal annotated datasets, along with the inherent noise and the diversity of multimodal relations in this type of data presents challenges for machine learning methods. Unlike classic multimodal data, social media data comes with a large diversity of relations between image and text making the interaction between the two modalities more difficult.Previous research concentrated on fusion strategies with separate encoders for each modality. This paper introduces CMB (Caption-based Multimodal BERT), a method of classifying crisis-related social media posts by utilizing information from both images and texts. CMB translates the image modality into a text-compatible space, facilitating intermodal interaction. Furthermore, CMB presents training opportunities to enhance the model's robustness to missing modalities. Experimental results show that CMB is competitive with well-established, costly, and manually crafted multimodal models.

Emotion-aware hierarchical interaction network for multimodal image aesthetics assessment

Image aesthetics assessment (IAA) has attracted increasing attention recently but is still challenging due to its high abstraction and complexity. Intuitively, image emotion and aesthetics are both human subjective feelings evoked by visual content. Previous researches have demonstrated that image aesthetics has intrinsic relationships with emotion. In fact, human emotional experience has potential impact on the perception of image aesthetics. Therefore, emotion information can be exploited to enhance aesthetic representation learning. Inspired by this, this paper presents an Emotion-Aware Hierarchical Interaction NETwork (EAHI-NET) for multimodal image aesthetics assessment, which explores both intra-modal and inter-modal interactions between aesthetics and emotions hierarchically. Specifically, we first propose the intra-modal emotion-aesthetics interaction module to obtain emotion-enhanced visual and textual aesthetic representations respectively. Then we propose the inter-modal feature enhancement to obtain the cross-modal aesthetic and emotion features. Finally, we design the inter-modal emotion-aesthetics interaction module to further investigate the cross-modal interplay between aesthetics and emotion, based on which hierarchical feature representations are achieved for multimodal IAA. The extensive experiments show that the proposed method can outperform the state-of-the-arts on multimodal AVA and Photo.net datasets.

Nonlinear scaling of fluctuation kinetic energy for shock–vorticity wave interaction

Turbulent kinetic energy (TKE) is a quantity of primary importance in shock–turbulence interaction (STI). Linear interaction analysis (LIA) serves as a theoretical method to predict the amplification of TKE in STI. LIA is constrained by low-amplitude fluctuations and neglects nonlinear effects. In this paper, we explore the nonlinear amplification of the kinetic energy of fluctuations in the interaction between a vorticity wave and a shock wave that serves as a building block flow for STI. A weakly nonlinear framework (WNLF) is introduced to analyze nonlinear effects in fluctuation kinetic energy (FKE) and identify the dominant physical mechanisms driving its amplification. The theoretical framework is validated through high-accuracy numerical simulations of shock–vorticity wave interactions. The simulation results are compared with the predictions derived from the WNLF for a range of intensities and inclinations of shock-upstream vorticity fluctuations at different Mach numbers, and WNLF is found to successfully scale the numerical data for FKE, thus confirming the validity and applicability of the framework. According to WNLF findings, at lower supersonic Mach numbers, the intermodal interaction between vorticity–vorticity modes is important, whereas the interaction between vorticity and acoustic modes becomes dominant at higher Mach numbers. Using the intermodal interactions, a model based on WNLF is proposed to predict TKE amplification in STI. In comparison to direct numerical simulation, the WNLF based model predicts the TKE amplification for moderate turbulent Mach number and lower supersonic flow Mach numbers. This is a significant improvement over the LIA results available in the literature.

DGL: Dynamic Global-Local Prompt Tuning for Text-Video Retrieval

Text-video retrieval is a critical multi-modal task to find the most relevant video for a text query. Although pretrained models like CLIP have demonstrated impressive potential in this area, the rising cost of fully finetuning these models due to increasing model size continues to pose a problem. To address this challenge, prompt tuning has emerged as an alternative. However, existing works still face two problems when adapting pretrained image-text models to downstream video-text tasks: (1) The visual encoder could only encode frame-level features and failed to extract global-level general video information. (2) Equipping the visual and text encoder with separated prompts failed to mitigate the visual-text modality gap. To this end, we propose DGL, a cross-modal Dynamic prompt tuning method with Global-Local video attention. In contrast to previous prompt tuning methods, we employ the shared latent space to generate local-level text and frame prompts that encourage inter-modal interaction. Furthermore, we propose modeling video in a global-local attention mechanism to capture global video information from the perspective of prompt tuning. Extensive experiments reveal that when only 0.67% parameters are tuned, our cross-modal prompt tuning strategy DGL outperforms or is comparable to fully finetuning methods on MSR-VTT, VATEX, LSMDC, and ActivityNet datasets. Code will be available at https://github.com/knightyxp/DGL.

DI-VTR: Dual inter-modal interaction model for video-text retrieval

Video-text retrieval is a challenging task for multimodal information processing due to the semantic gap between different modalities. However, most existing methods do not fully mine the intra-modal interactions, as with the temporal correlation of video frames, which results in poor matching performance. Additionally, the imbalanced semantic information between videos and texts also leads to difficulty in the alignment of the two modalities. To this end, we propose a dual inter-modal interaction network for video-text retrieval, i.e., DI-VTR. To learn the intra-modal interaction of video frames, we design a contextual-related video encoder to obtain more fine-grained content-oriented video representations. We also propose a dual inter-modal interaction module to accomplish accurate multilingual alignment between the video and text modalities by introducing multilingual text to improve the representation ability of text semantic features. Extensive experimental results on commonly-used video-text retrieval datasets, including MSR-VTT, MSVD and VATEX, show that the proposed method achieves significantly improved performance compared with state-of-the-art methods.

HATF: Multi-Modal Feature Learning for Infrared and Visible Image Fusion via Hybrid Attention Transformer

Current CNN-based methods for infrared and visible image fusion are limited by the low discrimination of extracted structural features, the adoption of uniform loss functions, and the lack of inter-modal feature interaction, which make it difficult to obtain optimal fusion results. To alleviate the above problems, a framework for multimodal feature learning fusion using a cross-attention Transformer is proposed. To extract rich structural features at different scales, residual U-Nets with mixed receptive fields are adopted to capture salient object information at various granularities. Then, a hybrid attention fusion strategy is employed to integrate the complementing information from the input images. Finally, adaptive loss functions are designed to achieve optimal fusion results for different modal features. The fusion framework proposed in this study is thoroughly evaluated using the TNO, FLIR, and LLVIP datasets, encompassing diverse scenes and varying illumination conditions. In the comparative experiments, HATF achieved competitive results on three datasets, with EN, SD, MI, and SSIM metrics reaching the best performance on the TNO dataset, surpassing the second-best method by 2.3%, 18.8%, 4.2%, and 2.2%, respectively. These results validate the effectiveness of the proposed method in terms of both robustness and image fusion quality compared to several popular methods.

YOLO-MS: Multispectral Object Detection via Feature Interaction and Self-Attention Guided Fusion

Object detection is essential for an autonomous driving sensing system. Since the light condition is changed in unconstrained scenarios, the detection accuracy based on visible images can be greatly degraded. Although the detection accuracy can be improved by fusing visible and infrared images, existing multispectral object detection (MOD) algorithms suffer from inadequate inter-modal interaction and a lack of global dependence in the fusion approach. Thus, we propose a multispectral object detection framework called YOLO-MS by designing a Feature Interaction and Self-Attention Fusion Network (FISAFN) as the backbone network. Within the FISAFN, correlations between two modalities are extracted by the Feature Interaction Module (FIM) for reconstructing the information components of each modality and enhancing capability of information exchange. To filter redundant features and enhance complementary features, long-range information dependence between two modalities are established by using a Self-Attention Feature Fusion Module (SAFFM). Thus, a better information richness of the fused features can be achieved. Experimental results on the FLIR-aligned dataset and the MFD dataset demonstrate that the proposed YOLO-MS performs favorably against state-of-the-art approaches, including feature-level fusion and pixel-level fusion. And further, the proposed YOLO-MS possesses good detection performance under diverse scene conditions.

GraphMFT: A graph network based multimodal fusion technique for emotion recognition in conversation

Multimodal machine learning is an emerging area of research, which has received a great deal of scholarly attention in recent years. Up to now, there are few studies on multimodal Emotion Recognition in Conversation (ERC). Since Graph Neural Networks (GNNs) possess the powerful capacity of relational modeling, they have an inherent advantage in the field of multimodal learning. GNNs leverage the graph constructed from multimodal data to perform intra- and inter-modal information interaction, which effectively facilitates the integration and complementation of multimodal data. In this work, we propose a novel Graph network based Multimodal Fusion Technique (GraphMFT) for emotion recognition in conversation. Multimodal data can be modeled as a graph, where each data object is regarded as a node, and both intra- and inter-modal dependencies existing between data objects can be regarded as edges. GraphMFT utilizes multiple improved graph attention networks to capture intra-modal contextual information and inter-modal complementary information. In addition, the proposed GraphMFT attempts to address the challenges of existing graph-based multimodal conversational emotion recognition models such as MMGCN. Empirical results on two public multimodal datasets reveal that our model outperforms the State-Of-The-Art (SOTA) approaches with the accuracy of 67.90% and 61.30%.

Medical visual question answering with symmetric interaction attention and cross-modal gating

The purpose of medical visual question answering (Med-VQA) is to provide accurate answers to clinical questions related to visual content of medical images. However, previous attempts neglect to take full advantage of the information interaction between medical images and clinical questions, which hinders the further progress of Med-VQA. The above issue requires the efforts to focus on critical information interaction within each modality and relevant information interaction between modalities. In this paper, we utilize the multiple meta-model quantifying model as visual encoder and the GloVe word embedding followed by the LSTM as textual encoder to form our feature extraction module. Then, we design a symmetric interaction attention module to construct dense and deep intra- and inter-modal information interaction on medical images and clinical questions for the Med-VQA task. Specifically, the symmetric interaction attention module consists of multiple symmetric interaction attention blocks that contain two basic units, i.e., self-attention and interaction attention. Technically, self-attention is introduced for intra-modal information interaction, while interaction attention is constructed for inter-modal information interaction. In addition, we develop a multi-modal fusion scheme that leverages the cross-modal gating to effectively fuse multi-modal information and avoid redundant information after sufficient intra- and inter-modal information interaction. Experimental results on the VQA-RAD dataset and PathVQA dataset show that our method outperforms other state-of-the-art Med-VQA models, achieving 74.7% and 48.7% on accuracy, 73.5% and 46.0% on F1-score, respectively.

Image-Text Retrieval With Cross-Modal Semantic Importance Consistency

Cross-modal image-text retrieval is an important area of Vision-and-Language task that models the similarity of image-text pairs by embedding features into a shared space for alignment. To bridge the heterogeneous gap between the two modalities, current approaches achieve inter-modal alignment and intra-modal semantic relationship modeling through complex weighted combinations between items. In the intra-modal association and inter-modal interaction processes, the higher-weight items have a higher contribution to the global semantics. However, the same item always produces different contributions in the two processes, since most traditional approaches only focus on the alignment. This usually results in semantic changes and misalignment. To address this issue, this paper proposes Cross-modal Semantic Importance Consistency (CSIC) which achieves invariance in the semantic of items during aligning. The proposed technique measures the semantic importance of items obtained from intra-modal and inter-modal self-attention and learns a more reasonable representation vector by inter-calibrating the importance distribution to improve performance. We conducted extensive experiments on the Flickr30K and MS COCO datasets. The results show that our approach can significantly improve retrieval performance, proving the proposed approach’s superiority and rationality.

Multimodal Emotion Analysis Model based on Interactive Attention Mechanism

In traditional multi-modal sentiment analysis, feature fusion is usually achieved by simple splicing, and multi-modal sentiment analysis is only trained as a single task, without considering the contribution of inter-modal information interaction to sentiment analysis and the correlation and constraint relationship between multi-modal and single-modal (text, video and audio) tasks. Therefore, a multi-task model based on interactive attention mechanism is proposed in this paper, which uses inter-modal attention mechanism and single-modal self-attention mechanism to train multi-modal sentiment analysis and single-modal sentiment analysis together, so as to make full use of inter-modal and inter-task information sharing, mutual complement, and reduce noise to improve the overall recognition performance. Experiments show that the proposed model performs well on MOSI and MOSEI common data sets for multimodal sentiment analysis.

Quaternion Representation Learning for cross-modal matching

The main challenge of cross-modal matching is to construct a shared subspace reflecting semantic closeness. Asymmetric relevance, especially the one-to-many matching case where multiple correspondences of a given query from the other modality can be obtained, has seriously exacerbated this difficulty. Recently, many approaches based on deep metric learning and probability distribution fitting have made great progress. However, these methods cannot generalize well to other approaches for their too many hyper-parameters or uncontrollable instability. In addition, we argue that the representational ability of common space learned in real space is insufficient and the symmetric similarity calculations adopted by the previous work fail to capture the asymmetric relevance. To seek a remedy for these problems, this work introduces a novel and effective approach called Quaternion Representation Learning (QRL) for better cross-modal matching. Specifically, benefiting from the strong expressive power and richer representation capability of quaternion space with three imaginary components, our proposed QRL method can better represent the shared semantic. Due to the inherent asymmetry of the Hamilton product and its latent inter-dependencies of components, our QRL method can model the asymmetric relevance and capture the complex similarity among intra- and inter-modal interaction. Another advantage is that our QRL method can be utilized in conjunction with other existing methods to promote cross-modal matching. Extensive experiments on two commonly used image-text matching benchmarks, i.e., MSCOCO and Flickr30K, and two widely used video-text retrieval datasets, including MSRVTT and TGIF, demonstrate the effectiveness and superiority of our proposed QRL method.

Cross-modal transformer with language query for referring image segmentation

Referring image segmentation (RIS) aims to predict a segmentation mask for a target specified by a natural language expression. However, the existing methods failed to implement deep interaction between vision and language is needed in RIS, resulting inaccurate segmentation. To address the problem, a cross-modal transformer (CMT) with language queries for referring image segmentation is proposed. First, a cross-modal encoder of CMT is designed for intra-modal and inter-modal interaction, capturing context-aware visual features. Secondly, to generate compact visual-aware language queries, a language-query encoder (LQ) embeds key visual cues into linguistic features. In particular, the combination of the cross-modal encoder and language query encoder realizes the mutual guidance of vision and language. Finally, the cross-modal decoder of CMT is constructed to learn multimodal features of the referent from the context-aware visual features and visual-aware language queries. In addition, a semantics-guided detail enhancement (SDE) module is constructed to fuse the semantic-rich multimodal features with detail-rich low-level visual features, which supplements the spatial details of the predicted segmentation masks. Extensive experiments on four referring image segmentation datasets demonstrate the effectiveness of the proposed method.

Multimodal Sentiment Recognition With Multi-Task Learning

Sentiment recognition in social network aims at recognizing the underlying affective states of user-generated content. The research center of sentiment recognition is moving from pure texts to multimodal contents, with the explosive growth of social platforms. Different from the image-text contents on blog and review platforms, multimodal sequences play a dominant role in streaming media, e.g. YouTube and TikTok. Sentiment recognition for multimodal sequences needs to extract the common and specific information of modalities. But current studies only focus on learning the cross-modal fusion representation to explore the inter-modal interaction, while neglecting the interactions and characteristics within each modality. We propose a novel cascade and specific scoring model, which aims at learning better cross-modal and unimodal representations to capture both the inter- and intra-modal interactions for sentiment recognition. Qualitative and quantitative experiments on two benchmarks have demonstrated the competitive performances of the proposed methods.

Nonlinear generation of sound and mean pressure drop in shock–shear interaction

Interactions of shear fluctuations with shock waves are ubiquitous in high-speed flow applications from scramjet propulsion to cosmic events like supernova explosions. They also serve as fundamental building blocks for the study of shock–turbulence interaction. In this work, we study the nonlinear effects in pressure arising due to the interaction of a normal shock with a two-dimensional shear wave. It employs the weakly nonlinear framework (WNLF) developed recently for vorticity amplification by Thakare et al. [“A weakly nonlinear framework to study shock–vorticity interaction,” J. Fluid Mech. 933, A48 (2022)]. The analysis includes the effect of intermodal interactions that is neglected in the widely used linear interaction analysis (LIA) of shock–turbulence interaction. It is found that the deformation of the shock wave and the fluctuation mass flux normal to the shock contribute to the dominant physical mechanisms responsible for the observed nonlinearities. Interestingly, the WNLF predicts a drop in mean pressure behind the shock due to a second-order intermodal interaction, which is consistent with the well-established results by Lele [“Shock-jump relations in a turbulent flow,” Phys. Fluids A 4, 2900–2905 (1992)] at low Mach numbers and brings out additional effects of shock deformation that are important at higher Mach numbers. We extend the WNLF to three-dimensional interaction of homogeneous isotropic turbulence with a normal shock. Comparison with existing direct numerical simulation data shows good agreement for low turbulent Mach numbers, which is a significant improvement over the prediction capability of LIA. We also compute the dilatation fields from WNLF and use them to distinguish between the acoustic and non-acoustic components of the second-order pressure fluctuations generated by the shock wave.

Exploiting enhanced and robust RGB-D face representation via progressive multi-modal learning

Existing RGB-based 2D face recognition approaches are sensitive to facial variations, posture, occlusions, and illumination. Current depth-based methods have been proved to alleviate the above sensitivity by introducing geometric information but rely heavily on high-quality depth from high-cost RGB-D cameras. To this end, we propose a Progressive Multi-modal Fusion framework to exploit enhanced and robust face representation for RGB-D facial recognition based on low-cost RGB-D cameras, which also deals with incomplete RGB-D modal data. Due to the defects such as holes caused by low-cost cameras, we first design a depth enhancement module to refine the low-quality depth and correct depth inaccuracies. Then, we extract and aggregate augmented feature maps of RGB and depth modality step-by-step. Subsequently, the masked modeling scheme and iterative inter-modal feature interaction module aim to fully exploit the implicit relations among these two modalities. We perform comprehensive experiments to verify the superior performance and robustness of the proposed solution over other FR approaches on four challenging benchmark databases.

Intermodal dispersive waves and soliton collision during multimode supercontinuum generation in chalcogenide glass fiber

Comparing with fused silica fiber, chalcogenide glass fibers are considered as an ideal platform for mid-infrared supercontinuum generation (SCG) because of its higher nonlinear coefficient and wider transparency window. Multimode fiber provides a new way to achieve special spectral characteristics in SCG by introducing intermodal interaction. We investigate the intermodal nonlinear phenomena associated with SCG in multimode chalcogenide fiber by numerically solving multimode generalized nonlinear Schrödinger equation. The results indicate that when the phase matching condition is satisfied, the intermodal interaction leads to the generation of dispersive waves at mid-infrared wavelength. Furthermore, intermodal and intramodal soliton collision are observed, and the bandwidth of the supercontinuum is increased by the intermodal interaction. These results may provide a new way for the generation of mid-infrared supercontinuum light source.

Tri-Modal Dense Video Captioning Based on Fine-Grained Aligned Text and Anchor-Free Event Proposals Generator

Multi-modal dense video captioning is a task using multiple information to detect all meaningful events and generate a textual description for each event. The existing works mainly rely on single visual or dual audio-visual modals in dense video captioning, while completely ignoring the text modal (subtitle). The text modal has a similar data structure as the video captions, which provides immediate semantic information to the content description for a video. In this paper, we propose a novel framework, called Two-Stage Cross-Modal Encoding Transformer Network (TS-CMETN), to realize the multi-modal dense video captioning task by fusing multiple features, including audio, visual, and text. First, we design a two-stage feature fusion encoder that hierarchically achieves the intra- and inter-modal information interaction. Second, we propose an anchor-free temporal event proposal module, which efficiently generates event proposals at each time step without the complex anchor calculation. Extensive experiments on the ActivityNet Captions dataset show that our proposed framework achieves high performance. Moreover, our approach can adaptively handle cases of the missing text modal. Our code and data are available at https://github.com/xieyulai/TM-CMETN .

Multimodal Sentiment Analysis Based on Cross-Modal Attention and Gated Cyclic Hierarchical Fusion Networks.

Multimodal sentiment analysis has been an active subfield in natural language processing. This makes multimodal sentiment tasks challenging due to the use of different sources for predicting a speaker's sentiment. Previous research has focused on extracting single contextual information within a modality and trying different modality fusion stages to improve prediction accuracy. However, a factor that may lead to poor model performance is that this does not consider the variability between modalities. Furthermore, existing fusion methods tend to extract the representational information of individual modalities before fusion. This ignores the critical role of intermodal interaction information for model prediction. This paper proposes a multimodal sentiment analysis method based on cross-modal attention and gated cyclic hierarchical fusion network MGHF. MGHF is based on the idea of distribution matching, which enables modalities to obtain representational information with a synergistic effect on the overall sentiment orientation in the temporal interaction phase. After that, we designed a gated cyclic hierarchical fusion network that takes text-based acoustic representation, text-based visual representation, and text representation as inputs and eliminates redundant information through a gating mechanism to achieve effective multimodal representation interaction fusion. Our extensive experiments on two publicly available and popular multimodal datasets show that MGHF has significant advantages over previous complex and robust baselines.

Cross-Modal Multitask Transformer for End-to-End Multimodal Aspect-Based Sentiment Analysis

As an emerging task in opinion mining, End-to-End Multimodal Aspect-Based Sentiment Analysis (MABSA) aims to extract all the aspect-sentiment pairs mentioned in a pair of sentence and image. Most existing methods of MABSA do not explicitly incorporate aspect and sentiment information in their textual and visual representations and fail to consider the different contributions of visual representations to each word or aspect in the text. To tackle these limitations, we propose a multi-task learning framework named Cross-Modal Multitask Transformer (CMMT), which incorporates two auxiliary tasks to learn the aspect/sentiment-aware intra-modal representations and introduces a Text-Guided Cross-Modal Interaction Module to dynamically control the contributions of the visual information to the representation of each word in the inter-modal interaction. Experimental results demonstrate that CMMT consistently outperforms the state-of-the-art approach JML by 3.1, 3.3, and 4.1 absolute percentage points on three Twitter datasets for the End-to-End MABSA task, respectively. Moreover, further analysis shows that CMMT is superior to comparison systems in both aspect extraction (AE) and sentiment classification (SC), which would move the development of multimodal AE and SC algorithms forward with improved performance.