Cross-modal Feature Research Articles

Cross-Modal Federated Human Activity Recognition.

Federated human activity recognition (FHAR) has attracted much attention due to its great potential in privacy protection. Existing FHAR methods can collaboratively learn a global activity recognition model based on unimodal or multimodal data distributed on different local clients. However, it is still questionable whether existing methods can work well in a more common scenario where local data are from different modalities, e.g., some local clients may provide motion signals while others can only provide visual data. In this article, we study a new problem of cross-modal federated human activity recognition (CM-FHAR), which is conducive to promote the large-scale use of the HAR model on more local devices. CM-FHAR has at least three dedicated challenges: 1) distributive common cross-modal feature learning, 2) modality-dependent discriminate feature learning, 3) modality imbalance issue. To address these challenges, we propose a modality-collaborative activity recognition network (MCARN), which can comprehensively learn a global activity classifier shared across all clients and multiple modality-dependent private activity classifiers. To produce modality-agnostic and modality-specific features, we learn an altruistic encoder and an egocentric encoder under the constraint of a separation loss and an adversarial modality discriminator collaboratively learned in hyper-sphere. To address the modality imbalance issue, we propose an angular margin adjustment scheme to improve the modality discriminator on modality-imbalanced data by enhancing the intra-modality compactness of the dominant modality and increase the inter-modality discrepancy. Moreover, we propose a relation-aware global-local calibration mechanism to constrain class-level pairwise relationships for the parameters of the private classifier. Finally, through decentralized optimization with alternative steps of adversarial local updating and modality-aware global aggregation, the proposed MCARN obtains state-of-the-art performance on both modality-balanced and modality-imbalanced data.

DeFuseDTI: Interpretable drug target interaction prediction model with dual-branch encoder and multiview fusion

Predicting the interaction between drugs and targets is a crucial step in drug development, and computer-based deep learning approaches have the potential to significantly reduce costs. Existing models using a single encoder often suffer from insufficient cross-modal feature extraction, with most models tending to overly focus on extracting locally aggregated information, thereby diluting the detailed features of each target residue and drug atom. Additionally, the lack of effective interaction fusion between drug and target lead to prediction results lacking reliable interpretability, posing a more urgent issue. To address these challenges, we propose a dual-branch encoder model, DeFuseDTI, which includes base encoder and detail encoder to extract locally aggregated features and detailed features of each target residue and drug atom. The detail encoder (utilizing Invertible Neural Networks for targets and graph transformers for drugs) can capture furtherly the features of each atom and residue, providing rich and precise features for model interpretability. For better achieve interactive learning of drug and target features, the Multiview Fusion Attention learning module was introduced to integrate multiview features and generate a unified representations for decoding prediction results. Based on the module's unique attention mechanism, drug-target importance matrices can be obtained, which offer interpretable analysis at the molecular level. Experimental results and analyses demonstrate that DeFuseDTI outperforms several state-of-the-art models on four public datasets. Its significant interpretability holds promise for providing accurate and scientifically meaningful guidance for biochemical experiments at the molecular level.

CFINet: Cross-Modality MRI Feature Interaction Network for Pseudoprogression Prediction of Glioblastoma.

Pseudoprogression (PSP) is a related reaction of glioblastoma treatment, and misdiagnosis can lead to unnecessary intervention. Magnetic resonance imaging (MRI) provides cross-modality images for PSP prediction studies. However, how to effectively use the complementary information between the cross-modality MRI to improve PSP prediction is still a challenging task. To address this challenge, we propose a cross-modality feature interaction network for PSP prediction. Firstly, we propose a triple-branch multi-scale module to extract low-order feature representations and a skip-connection multi-scale module to extract high-order feature representations. Then, a cross-modality interaction module based on attention mechanism is designed to make the complementary information between cross-modality MRI fully interact. Finally, the high-order cross-modality interaction information is fed into a multi-layer perceptron to achieve the PSP prediction task. We evaluate the proposed network on a private dataset with 52 subjects from Hunan Cancer Hospital and validate it on a private dataset with 30 subjects from Xiangya Hospital. The accuracy of our proposed network on the datasets is 0.954 and 0.929, respectively, which is better than most typical convolutional neural network and interaction methods.

Efficient cross-modality feature interaction for multispectral armored vehicle detection

Detecting armed vehicles from a UAV platform is challenging due to the complexity of ground environment. This paper presents a dual-stream multispectral armored vehicle detection method to tackle this problem. First, considering that there is a paucity of datasets containing multispectral armored vehicle images, a multispectral armored vehicle detection dataset is constructed for this study. The dataset consists of 5853 pairs of RGB and infrared images, featuring a total of 15,878 instances of armored vehicles. Then, a cross-modal feature interaction module is designed to enable efficient feature interaction between multispectral images. This module uses the cross-modal channel-wise feature difference method to model the channel differences between the two modal features and obtains the cross-modal channel difference matrix. The cross-modal channel difference matrix is then employed to extract the unique features of the two modal features, allowing for efficient cross-modal feature interaction by complementing each other's unique features. Experiment results demonstrate that the proposed model has excellent detection performance and is capable of coping with various challenges brought by complex ground environments.

Mitigating Modality Discrepancies for RGB-T Semantic Segmentation.

Semantic segmentation models gain robustness against adverse illumination conditions by taking advantage of complementary information from visible and thermal infrared (RGB-T) images. Despite its importance, most existing RGB-T semantic segmentation models directly adopt primitive fusion strategies, such as elementwise summation, to integrate multimodal features. Such strategies, unfortunately, overlook the modality discrepancies caused by inconsistent unimodal features obtained by two independent feature extractors, thus hindering the exploitation of cross-modal complementary information within the multimodal data. For that, we propose a novel network for RGB-T semantic segmentation, i.e. MDRNet+, which is an improved version of our previous work ABMDRNet. The core of MDRNet+ is a brand new idea, termed the strategy of bridging-then-fusing, which mitigates modality discrepancies before cross-modal feature fusion. Concretely, an improved Modality Discrepancy Reduction (MDR+) subnetwork is designed, which first extracts unimodal features and reduces their modality discrepancies. Afterward, discriminative multimodal features for RGB-T semantic segmentation are adaptively selected and integrated via several channel-weighted fusion (CWF) modules. Furthermore, a multiscale spatial context (MSC) module and a multiscale channel context (MCC) module are presented to effectively capture the contextual information. Finally, we elaborately assemble a challenging RGB-T semantic segmentation dataset, i.e., RTSS, for urban scene understanding to mitigate the lack of well-annotated training data. Comprehensive experiments demonstrate that our proposed model surpasses other state-of-the-art models on the MFNet, PST900, and RTSS datasets remarkably.

A Retinal Vessel Segmentation Network Fusing Cross-Modal Features

A Retinal Vessel Segmentation Network Fusing Cross-Modal Features

Dual-Pathway Deep Hashing-Based Adversarial Learning for Cross-Modal Retrieval

The application of deep hashing methods for cross-modal retrieval has seen growing interest due to their storage efficiency and fast query execution. However, the challenge posed by the “heterogeneity gap” in multi-modal datasets cannot be understated. To address this, we present a novel framework named Dual-Pathway Deep Hashing-Based Adversarial Learning (DP-DHAL), engineered to surmount this challenge. The architecture of DP-DHAL integrates three key components: (a) a dual-pathway representation learning module tasked with extracting modality-specific features; (b) an adversarial module working to align the distributions of cross-modal features; and (c) a deep hashing module responsible for generating hash codes that uphold the similarity relationships across different modalities. Additionally, we have developed a unique Hamming triplet-margin loss function to refine the assessment of content similarities. The DP-DHAL model is trained through an adversarial process where the adversarial module’s goal is to discern cross-modal features with the aim of reducing the heterogeneity gap. Simultaneously, the representation learning module is focused on producing representations that can both deceive the adversarial module and preserve cross-modal similarities to yield distinctive hash codes. Comprehensive experiments on varied datasets have shown that our proposed method outperforms other leading cross-modal hashing techniques.

Heterogeneous Interactive Graph Network for Audio–Visual Question Answering

Audio–visual question answering (AVQA) is an emerging task that aims to provide answers by integrating visual contents, audio streams, and their associations within given videos. The major challenge lies in effectively fusing heterogeneous multi-modal data to comprehend complex scenes while capturing question-related clues to infer correct answers. Current AVQA models primarily employ attention mechanisms to extract question-related clues separately from visual and audio modalities before combining them. However, these approaches have two limitations: (1) They neglect the exploration of the association and complementary between audio and visual; (2) Encoding visual or audio holistically limits the capacity to capture the cross-modal and cross-temporal dynamic events. In this paper, we introduce the Heterogeneous Interactive Graph Network, a novel solution designed to address these limitations. Specifically, we construct heterogeneous multi-modal graphs that facilitate unified integration of multiple modalities, including visual, audio, and question. This approach effectively explores the associations and complementarity among multiple modalities, and it investigates local temporal interactions across visual and audio, enabling the effective capture of cross-modal and cross-temporal dynamic events. Additionally, we present a cross-modal feature alignment module, which acts as a bridge to overcome the semantic gap among heterogeneous multi-modal data. It promotes the convergence of multi-modal data distributions into a shared feature space, facilitating more effective and efficient processing. Extensive experimental results demonstrate the superiority of our method compared to state-of-the-art models across various question types on the challenging MUSIC-AVQA and AVQA benchmarks.

Token-word mixer meets object-aware transformer for referring image segmentation

Referring image segmentation aims to generate a binary mask of the target object according to a referring expression. Some recent works argue that post-fusion paradigm may result in inconsistency and insufficiency issue and propose to integrate textual features during the visual encoding process. Although effective, they do fusion in a single way at each stage of encoder, e.g. utilizing cross attention mechanisms. This single fusion method ignores local and detailed image information correlated with language due to the incapability of attention in capturing high-frequencies information. To address this issue, we propose a Token-Word Mixer, which takes into consideration the characteristics of convolution and attention, and achieves more comprehensive interactions and alignments of multi-modal features through a mix operation. Furthermore, existing methods that rely solely on grid features lack perception of the target object and inference of relationships between objects, making it difficult to associate and align semantic information of target objects during multi-modal fusion when referring expressions or image scenes are complex. Therefore, we propose to incorporates object-level information by exploiting a DETR-based detector to provide region features, and the Object-Aware Transformer encoder with an additional learnable token is proposed to perceive effective information associated with the target object. Based on the enhanced cross-modal features and the aggregated token, we adopt query-based mask generation method instead of pixel classification framework for referring image segmentation. Extensive experiments and ablation studies indicate the effectiveness of our proposed methods.

CrowdAlign: Shared-weight dual-level alignment fusion for RGB-T crowd counting

The combination of visible and thermal images has been proven to be effective in improving accuracy for crowd counting in illumination-unconstrained scenes. However, the challenging problem of misalignment in RGB-T image pairs has not been extensively explored in this context. This study aims to address the issue of misalignment between RGB and thermal image pairs to enhance the counting accuracy of cross-modal models. Specifically, we propose CrowdAlign, a cross-modal feature alignment fusion network that utilizes a shared-weight strategy for efficient feature extraction. Additionally, CrowdAlign addresses alignment adjustments through two stages: pre-fusion and post-fusion alignment. For pre-fusion feature extraction, we design a dual-level spatial-semantic parallel alignment module, while for post-fusion feature extraction, a low-frequency feature attention filtering module is developed. This two-stage alignment approach enables cross-modal feature alignment without requiring additional supervision. Experiments on the public benchmarks demonstrate that our method is effective under RGB-T misalignment or dark conditions. We hope CrowdAlign will inspire researchers to focus on and explore the issue of misalignment between RGB image and thermal image for cross-modal crowd counting.

Object Detection in Multispectral Remote Sensing Images Based on Cross-Modal Cross-Attention.

In complex environments a single visible image is not good enough to perceive the environment, this paper proposes a novel dual-stream real-time detector designed for target detection in extreme environments such as nighttime and fog, which is able to efficiently utilise both visible and infrared images to achieve Fast All-Weatherenvironment sensing (FAWDet). Firstly, in order to allow the network to process information from different modalities simultaneously, this paper expands the state-of-the-art end-to-end detector YOLOv8, the backbone is expanded in parallel as a dual stream. Then, for purpose of avoid information loss in the process of network deepening, a cross-modal feature enhancement module is designed in this study, which enhances each modal feature by cross-modal attention mechanisms, thus effectively avoiding information loss and improving the detection capability of small targets. In addition, for the significant differences between modal features, this paper proposes a three-stage fusion strategy to optimise the feature integration through the fusion of spatial, channel and overall dimensions. It is worth mentioning that the cross-modal feature fusion module adopts an end-to-end training approach. Extensive experiments on two datasets validate that the proposed method achieves state-of-the-art performance in detecting small targets. The cross-modal real-time detector in this study not only demonstrates excellent stability and robust detection performance, but also provides a new solution for target detection techniques in extreme environments.

Multi-level Symmetric Semantic Alignment Network for image–text matching

Image–text matching has attracted much attention as one of the visual-linguistic tasks. Most of the existing methods tend to concentrate on single-level semantic similarity by global embeddings or local alignment, which fail to distinguish the ambiguous image–text instances with highly similar contexts due to insufficient interactions of cross-modal features. To mitigate this problem, in this paper, we propose a novel Multi-level Symmetric Semantic Alignment Network (MSSAN), which fully exploits the multi-level semantic features to capture the complicated correlations between images and text. Specifically, global–global-level alignment is first performed based on visual–textual global representations. Then, considering the local semantic consistency, local–local-level alignment is carried out through inter-modal bidirectional cross-attention module to model fine-grained region–word relations. Moreover, a Multi-granularity Feature Fusion Module (MFFM) is constructed to learn intra-modal distribution of significance and incorporate global concepts into local features to obtain more comprehensive semantic representations, thus achieving global–local-level alignment. Finally, in order to achieve a strong separation of semantically similar samples, we develop a novel generic triplet ranking loss with adaptively updatable margins to train our model. Extensive experimental results on Flickr8K, MSCOCO and Flickr30K datasets demonstrate that our proposed MSSAN is superior to other state-of-the-art methods by a considerable margin.

Cross-modal collaborative feature representation via Transformer-based multimodal mixers for RGB-T crowd counting

RGB-T crowd counting aims to utilize the thermal source data to compensate for the insufficient RGB feature representation under low-illumination conditions. However, how to efficiently use complementary information from different modalities is still a challenging issue for the target task. Existing CNN-based RGB-T crowd counting methods focus on the cross-modal feature representation via the local fixed-receptive-field convolutional operations and the transformer-based works mainly utilize the mutual attention, failing to realize the global cross-modal feature representation. To this end, this paper designs a multimodal counting network based on the multimodal transformer mixer to realize cross-modal collaborative feature representation. Firstly, we design a Transformer-based multimodal mixer to fully fuse the features from both modalities. And then we utilize the Transformer-based network with the multi-head self-attention layer replaced by the average pooling layer to build the two-stream backbone to extract rich multi-scale feature information of the two modalities. Meanwhile, the features extracted from the mixers are used to enhance the intermediate features to achieve the preservation of crowd detail information and suppression of background information. Finally, we design a pyramid regression head to aggregate the multi-scale feature maps to regress the density maps. Extensive experiments, as well as ablation studies, are conducted on challenging RGB-T crowd counting benchmark datasets (RGBT-CC and DroneRGBT), and the achieved competitive experimental results demonstrate the effectiveness of the proposed method on cross-modal collaborative feature representation for target task.

MAGNet: Multi-scale Awareness and Global fusion Network for RGB-D salient object detection

In recent years, excellent RGB-D salient object detection performance has been achieved. However, existing detection methods generally require a large number of model parameters in pursuit of high accuracy. To alleviate this problem, we propose a Multi-scale Awareness and Global fusion Network for RGB-D salient object detection, named MAGNet. MAGNet has 16.1M Params and 9.9G FLOPs. Specifically, we notice that convolutional neural networks (CNNs) can strongly perceive local spatial structures, whereas attention mechanisms can perform global correlation analysis of input information. Therefore, we exploit the advantages of both methods to design two kinds of cross-modal feature fusion modules. To reduce the computational complexity of the model, we design a multi-scale awareness fusion module (MAFM) to fully leverage the rich textural information and edge information in low-level feature maps. For the high-level feature maps, we incorporate an attention mechanism and a CNN to design a global fusion module (GFM), which enables the model to better capture the semantic information of different modalities by learning the correspondence between RGB and depth images. Then, we employ the proposed multi-level convolution module (MCM) to generate the predicted map through a step-by-step decoding process, which can gradually recover finer detection results. Finally, extensive experimental results on six datasets show that the proposed MAGNet not only achieves advanced detection performance but also drastically reduces the number of model parameters. Source code is available at https://github.com/mingyu6346/MAGNet.

Cross-modal learning for optical flow estimation with events

Benefiting from the low latency and high dynamic range, event cameras have recently been adopted for Optical Flow (OF) prediction under harsh environments with high-speed motion or extreme lighting conditions. However, the emitted events only respond to brightness changes, mostly at edges with high-intensity contrast, leading to the ill-posedness of OF prediction in non-edge or low-contrast regions. To address this problem, we propose a network to learn OF by exploring the complements between the frame and the event domains. In particular, we first extract cross-modal features from frames and events via a fusion-based encoder and then apply a self-attention module of cross-dimension interaction to suppress disturbances in noisy events. Furthermore, a long-term memory recurrent module is built for local and global temporal information combination and cross-modal feature enhancement. Finally, a CNN decoder is designed to upscale cross-modal features and provide pyramidal results for multi-scale loss computation. Extensive experiments demonstrate that the proposed method can achieve leading performance compared with the state-of-the-art OF models.

Enhancing target detection accuracy through cross-modal spatial perception and dual-modality fusion

The disparity between human and machine perception of spatial information presents a challenge for machines to accurately sense their surroundings and improve target detection performance. Cross-modal data fusion emerges as a potential solution to enhance the perceptual capabilities of systems. This article introduces a novel spatial perception method that integrates dual-modality feature fusion and coupled attention mechanisms to validate the improvement in detection performance through cross-modal information fusion. The proposed approach incorporates cross-modal feature extraction through a multi-scale feature extraction structure employing a dual-flow architecture. Additionally, a transformer is integrated for feature fusion, while the information perception of the detection system is optimized through the utilization of a linear combination of loss functions. Experimental results demonstrate the superiority of our algorithm over single-modality target detection using visible images, exhibiting an average accuracy improvement of 30.4%. Furthermore, our algorithm outperforms single-modality infrared image detection by 3.0% and comparative multimodal target detection algorithms by 3.5%. These results validate the effectiveness of our proposed algorithm in fusing dual-band features, significantly enhancing target detection accuracy. The adaptability and robustness of our approach are showcased through these results.

RagBERT: Relationship-aligned and grammar-wise BERT model for image captioning

Image captioning has become one of the most popular research problems in the field of artificial intelligence. Although many studies have achieved excellent results, there still are some challenges, for example, cross-modal feature alignment lacks explicit guidance, and model-generated sentences contain grammatical errors. In this paper, we propose a relationship-aligned and grammar-wise BERT model, which integrates a relationship exploration module and a grammar enhancement module into the BERT-based model. Specifically, in the relationship exploration module, to explore relationship tags as anchors to guide semantic alignment, we design a network to calculate the cosine similarity between visual features and word vector information. We construct the grammar enhancement module similarly to the BERT. That means we use two BERT modules in our framework. The first is the main frame for generating captions, and the second is the auxiliary model to determine whether the syntax of the generated caption is correct. To validate the performance of our proposed model, we conduct abundant experiments on the MSCOCO dataset, Flickr30k dataset, and Flickr8k dataset. Experimental results show that our proposed method performs better than state-of-the-art approaches.

AITFuse: Infrared and visible image fusion via adaptive interactive transformer learning

Existing deep learning-based methods often follow either image-level or feature-level fusion frameworks to uniformly or separately extract features, ignoring the specialized interactive information learning, which may produce limited fusion performance. To tackle this challenge, we devise a powerful fusion baseline via adaptive interactive Transformer learning, namely AITFuse. Unlike previous methods, our network alternately incorporates local and global relationships through collaborative learning of both CNN and Transformer. In particular, we propose a cascaded token-wise and channel-wise Vision Transformer architecture with different attention mechanisms to model the long-range contexts, and allow feature communication across different tokens and independent channels in an interactive manner. On this basis, the modal-specific feature rectification module employs self-attention operation to revise distinctive features within the same domain for efficient encoding. Meanwhile, the cross-modal feature integration module constructs cross-attention mechanism to fuse complementary characteristics from different domains for multi-level decoding. In addition, we discard the learning position embedding to release our fusion model for the image of arbitrary sizes without splitting operations. Extensive experiments on mainstream datasets and downstream tasks demonstrate the rationality and superiority of our AITFuse. The codes will be available at https://github.com/Zhishe-Wang/AITFuse.

High-level vision task-driven infrared and visible image fusion approach: Progressive Semantic Enhancement based Multi-scale Cross-modality Interactive Network

Infrared-visible image fusion aims to merge data for a more accurate scene representation. Current research focuses mainly on enhancing visual appeal rather than improving performance in high-level vision tasks. To address this gap, we propose the Semantic Enhanced Multi-scale Cross-modality Interactive Image Fusion Network (SeMIFusion). Initially, Multi-scale Cross-modality Feature Fusion (MCFF) module is devised to extract shallow and deep features across different modalities. During feature extraction, Texture Enhancer (TE) and Semantify Enhancer (SE) blocks capture diverse hierarchical features across multi-scale layers, seamlessly integrating into the Semantic Feature Integration (SFI) module for profound semantic information extraction. Furthermore, an Image Scene Reconstruction (ISR) module maintains original image details in fused features, ensuring image fidelity. Additionally, incorporating a visual preservation guiding mask prioritizes retaining visual quality during reconstruction, preventing degradation. Extensive experiments demonstrate our method’s superiority in preserving visual effects and texture details, especially in high-level vision tasks. The source code is available at https://github.com/yyzzttkkjj/SeMIFusion.

Video multimodal sentiment analysis using cross-modal feature translation and dynamical propagation

Multimodal sentiment analysis on social platforms is crucial for comprehending public opinions and attitudes, thus garnering substantial interest in knowledge engineering. Existing methods like implicit interaction, explicit interaction, and cross-modal translation can effectively integrate sentiment information, but they encounter challenges in establishing efficient emotional correlations across modalities due to data heterogeneity and concealed emotional relationships. To tackle this issue, we propose a video multimodal sentiment analysis model called PEST, which leverages cross-modal feature translation and a dynamic propagation model. Specifically, cross-modal feature translation translates textual, visual, and acoustic features into a common feature space, eliminating heterogeneity and enabling initial modal interaction. Additionally, the dynamic propagation model facilitates in-depth interaction and aids in establishing stable and reliable emotional correlations across modalities. Extensive experiments on the three multimodal sentiment datasets, CMU-MOSI, CMU-MOSEI, and CH-SIMS, demonstrate that PEST exhibits superior performance in both word-aligned and unaligned settings.