Cross-modal Data Research Articles

Bidirectional Attentional Interaction Networks for RGB-D salient object detection

Aiming at the issues of insufficient cross-modality feature interaction and ineffective utilization of cross-modality data in RGB-D saliency object detection (SOD) tasks, we propose a Bidirectional Attentional Interaction Network (BAINet) for RGB-D SOD, which employs an encoder-decoder structure for bidirectional interaction of cross-modality features through a dual-branch progressive fusion approach. To begin with, based on the fact that RGB and depth information streams can complement each other, the bidirectional attention interaction module accomplishes bidirectional interaction between cross-modality features by capturing complementary cues from different modality data. In order to enhance the expressiveness of the fused RGB-D features, the global feature perception module endows the features with rich multi-scale contextual semantic information by enlarging the field of perception. In addition, exploring the correlation of cross-level features is vital to achieve accurate salient inference. Specifically, We introduce a cross-level guidance aggregation module to capture inter-layer dependencies and complete the integration of cross-level features, which effectively suppresses shallow cross-modality features and refines the saliency map during decoding. To improve the model training speed, a hybrid loss function is employed to train multi-branch saliency inference maps simultaneously. Extensive experiments on five publicly available datasets clearly show that the proposed model outperforms 18 state-of-the-art methods.

Cross on Cross Attention: Deep Fusion Transformer for Image Captioning

Numerous studies have shown that in-depth mining of correlations between multi-modal features can help improve the accuracy of cross-modal data analysis tasks. However, the current image description methods based on the encoder-decoder framework only carry out the interaction and fusion of multi-modal features in the encoding stage or the decoding stage, which cannot effectively alleviate the semantic gap. In this paper, we propose a Deep Fusion Transformer (DFT) for image captioning to provide a deep multi-feature and multi-modal information fusion strategy throughout the encoding to decoding process. We propose a novel global cross encoder to align different types of visual features, which can effectively compensate for the differences between features and incorporate each other’s strengths. In the decoder, a novel cross on cross attention is proposed to realize hierarchical cross-modal data analysis, extending complex cross-modal reasoning capabilities through the multi-level interaction of visual and semantic features. Extensive experiments conducted on the MSCOCO dataset prove that our proposed DFT can achieve excellent performance and outperform state-of-the-art methods. The code is available at https://github.com/weimingboya/DFT.

Transformer-based cross-modal multi-contrast network for ophthalmic diseases diagnosis

Automatic diagnosis of various ophthalmic diseases from ocular medical images is vital to support clinical decisions. Most current methods employ a single imaging modality, especially 2D fundus images. Considering that the diagnosis of ophthalmic diseases can greatly benefit from multiple imaging modalities, this paper further improves the accuracy of diagnosis by effectively utilizing cross-modal data. In this paper, we propose Transformer-based cross-modal multi-contrast network for efficiently fusing color fundus photograph (CFP) and optical coherence tomography (OCT) modality to diagnose ophthalmic diseases. We design multi-contrast learning strategy to extract discriminate features from cross-modal data for diagnosis. Then channel fusion head captures the semantically shared information across different modalities and the similarity features between patients of the same category. Meanwhile, we use a class-balanced training strategy to cope with the situation that medical datasets are usually class-imbalanced. Our method is evaluated on public benchmark datasets for cross-modal ophthalmic disease diagnosis. The experimental results demonstrate that our method outperforms other approaches. The codes and models are available at https://github.com/ecustyy/tcmn.

Development and validation of an fMRI-informed EEG model of reward-related ventral striatum activation

Reward processing is essential for our mental-health and well-being. In the current study, we developed and validated a scalable, fMRI-informed EEG model for monitoring reward processing related to activation in the ventral-striatum (VS), a significant node in the brain's reward system. To develop this EEG-based model of VS-related activation, we collected simultaneous EEG/fMRI data from 17 healthy individuals while listening to individually-tailored pleasurable music – a highly rewarding stimulus known to engage the VS. Using these cross-modal data, we constructed a generic regression model for predicting the concurrently acquired Blood-Oxygen-Level-Dependent (BOLD) signal from the VS using spectro-temporal features from the EEG signal (termed hereby VS-related-Electrical Finger Print; VS-EFP). The performance of the extracted model was examined using a series of tests that were applied on the original dataset and, importantly, an external validation dataset collected from a different group of 14 healthy individuals who underwent the same EEG/FMRI procedure. Our results showed that the VS-EFP model, as measured by simultaneous EEG, predicted BOLD activation in the VS and additional functionally relevant regions to a greater extent than an EFP model derived from a different anatomical region. The developed VS-EFP was also modulated by musical pleasure and predictive of the VS-BOLD during a monetary reward task, further indicating its functional relevance. These findings provide compelling evidence for the feasibility of using EEG alone to model neural activation related to the VS, paving the way for future use of this scalable neural probing approach in neural monitoring and self-guided neuromodulation.

Cross-modal Person Re-identification Based on Hybrid Learning Networks

Objective: Person re-identification (cross-modal person re-identification, Re-ID) aims to retrieve query identities from a gallery of non-overlapping camera views. It has many practical applications such as missing person search, forensics, etc. Much pedestrian Re-ID research mainly focuses on single-modal retrieval tasks, where both indoor and outdoor pedestrian images are from red, green, and blue (RGB) modalities. However, the security monitoring system will automatically switch from the visible state to the infrared state in dark situations. In order to meet the needs of practical applications, research on cross-modal pedestrian Re-ID based on IR-RGB has attracted more and more attention. The features of different modality images essentially lack modality sharing information. Methods: Most of the existing methods use the network to extract shared features to deal with differences, but they ignore the unique feature information that is beneficial to recognition in these two modality images. In this paper, a hybrid learning network is proposed to solve the problem of learning the unique and common features of the two modalities by mixing the single-modal branch and the cross-modal branch to learn from each other. At the same time, the graph convolution attention mechanism and granular feature learning are used to process the features extracted by the cross-modal branch and the single-modal branch, so as to enrich the information input by different branch classifiers, thereby improving the discriminability of features. Results: And achieved 59.68% Rank-1 accuracy and 58.28% mean average precision average mean precision on the cross-modal data set SYSU-MM01. Conclusion: The experimental results also show that this is the effectiveness of the method.

Semantic-embedding Guided Graph Network for cross-modal retrieval

Many methods focus on aligning image regions with the corresponding text fragments, and ignore that images contain fragments that cannot be expressed by texts. To fully express the information of images and prevent the performance degradation caused by fine-grained information deviating from the core meaning of images, we propose Semantic-embedding Guided Graph Network (SGGN) for cross-modal retrieval. It learns the detail representations of each modality, with an integrated semantic information, by guiding local fragments to capture the internal correlation of cross-modal data and effectively convey the information. To further bridge the semantic gap between different modalities, SGGN uses adversarial network to play a game, and uses graph aggregation network to absorb complementary information of neighbor samples. We evaluate our approach on two datasets. Our method (based on R@10) achieves 97.2% on Flickr30k dataset. On MS-COCO dataset, it reaches 99.2% using 1K test set and 92.0% using 5K test set.

Contrastive learning-based general Deepfake detection with multi-scale RGB frequency clues

Deepfake is a type of image and video face manipulation methods which could cause security and society threats. Although some related databases and detection models have been proposed for detecting face forgery media, achieving a generalizable detector for both known and unknown manipulations remains challenging. In this study, a novel deepfake detection model with high generalizability is proposed to tackle this issue. We employ supervised contrastive learning to enhance the generalizability to unknown manipulations and datasets. In addition, we design a cross-modality data augmentation method by combining SRM and RGB features to extract detection clues comprehensively. Furthermore, we propose a multi-scale feature enhancement module to enhance textural and semantic information. Extensive experiments have demonstrated that our method improves model generalization in both intra- and cross- dataset scenarios.

Semantic Completion and Filtration for Image–Text Retrieval

Image–text retrieval is a vital task in computer vision and has received growing attention, since it connects cross-modality data. It comes with the critical challenges of learning unified representations and eliminating the large gap between visual and textual domains. Over the past few decades, although many works have made significant progress in image–text retrieval, they are still confronted with the challenge of incomplete text descriptions of images, i.e., how to fully learn the correlations between relevant region–word pairs with semantic diversity. In this article, we propose a novel semantic completion and filtration (SCAF) method to alleviate the above issue. Specifically, the text semantic completion module is presented to generate a complete semantic description of an image using multi-view text descriptions, guiding the model to explore the correlations of relevant region–word pairs fully. Meanwhile, the adaptive structural semantic matching module is presented to filter irrelevant region–word pairs by considering the relevance score of each region–word pair, which facilitates the model to focus on learning the relevance of matching pairs. Extensive experiments show that our SCAF outperforms the existing methods on Flickr30K and MSCOCO datasets, which demonstrates the superiority of our proposed method.

Cardiac LGE MRI Segmentation With Cross-Modality Image Augmentation and Improved U-Net.

Image segmentation is a challenging problem in imaging informatics, which stems from the intersection of imaging techniques, computer science and biomedicine. In particular, accurate segmentation of cardiac structures in late gadolinium enhancement (LGE) cardiac magnetic resonance (CMR) is of great clinical importance for cardiac function assessment and myocardial disease diagnosis. However, it is a well-known challenge due to its special imaging modality and the lack of labeled LGE samples. In this paper, we propose an unsupervised ventricular segmentation algorithm that can perform biventricular segmentation of LGE images in the absence of labeled LGE data. There are two primary modules, the data augmentation procedure and the segmentation network. The easily available annotated balanced-Steady State Free Precession (bSSFP) images are employed for cross-modal data augmentation by image translation, where a single bSSFP image is converted into multiple synthetic LGE images while preserving the original morphological structure. Then, the proposed segmentation network is trained with the synthetic LGE images and used for segmenting real LGE images. Validation experiments demonstrated the effectiveness and advantages of the proposed algorithm.

BCINet: Bilateral cross-modal interaction network for indoor scene understanding in RGB-D images

Depth cue has proven to be useful information in the indoor scene understanding of RGB-D images for providing a geometric counterpart to RGB representation. However, because of the differences between RGB-D image pairs, utilizing cross-modal data effectively is a key issue. Most methods exclusively leverage depth data to unilaterally complement RGB data for better feature representation; they invariably ignore the fact that RGB and depth data can bilaterally complement each other. Herein, a novel RGB-D scene-understanding network called BCINet is presented, in which RGB and depth data bilaterally complement each other via a proposed bilateral cross-modal interaction module (BCIM). The BCIM helps to capture cross-modal complementary cues by crossly fusing enhanced features from one modality to the counterpart modality through a feature enhanced module. Meanwhile, exploiting the long-range dependencies of RGB-D features is also significant for accurate scene understanding. Specifically, we design a hybrid pyramid dilated convolution module to enlarge the receptive fields along both the vertical and horizontal spatial directions to adaptively capture diverse contexts with different shapes. Additionally, we propose a context-guided module to aggregate these diverse higher-level contexts with lower-level features in the encoder to guide the information flow for progressively refining the segmentation map. Experimental results on two indoor scene datasets demonstrate the superiority and effectiveness of the proposed BCINet over several state-of-the-art approaches.

Unsupervised Cross-Media Graph Convolutional Network for 2D Image-Based 3D Model Retrieval

With the rapid development of 3D construction technology, 3D models have been implemented in many applications. In particular, the fields of virtual and augmented reality have created a considerable demand for rapid access to large sets of 3D models in recent years. An effective method for addressing the demand is to search 3D models based on 2D images because 2D images can be easily captured by smartphones or other lightweight vision sensors. In this paper, we propose a novel unsupervised cross-media graph convolutional network (UCM-GCN) for 3D model retrieval based on 2D images. Here, we render views from 3D models to construct a graph model based on 3D model structural information. Then, we utilize the 2D image's visual information to bridge the gap between cross-modality data. Then, the proposed UCM-GCN is utilized to update the feature vector of the 2D image and the 3D model. Here, we introduce correlation loss to mitigate the distribution discrepancy across different modalities, which can fully consider the structural and visual similarities between the 2D image and 3D model to embed the final different modalities into the same feature space. To demonstrate the performance of our approach, we conducted a series of experiments on the MI3DOR dataset, which is utilized in SHREC19. We also compared it with other similar methods on the 3D-FUTURE dataset. The experimental results demonstrate the superiority of our proposed method over state-of-the-art methods.

Interacting-Enhancing Feature Transformer for Cross-Modal Remote-Sensing Image and Text Retrieval

Cross-modal remote sensing image-text retrieval (CMRSITR) is a challenging topic in the remote sensing (RS) community. It has gained growing attention because it can be flexibly used in many practical applications. In the current deep era, with the help of deep convolutional neural networks (DCNNs), many successful CMRSITR methods have been proposed. Most of them first learn valuable features from RS images and texts respectively. Then, the obtained visual and textual features are mapped into a common space for the final retrieval. The above operations are feasible, however, two difficulties are still to be solved. One is that the semantics within the visual and textual features are misaligned due to the independent learning manner. The other one is that the deep links between RS images and texts cannot be fully explored by simple common space mapping. To overcome the above challenges, we propose a new model named interacting-enhancing feature transformer (IEFT) for CMRSITR, which regards the RS images and texts as a whole. First, a simple feature embedding module (FEM) is developed to map images and texts into the visual and textual feature spaces. Second, an information interacting-enhancing module (IIEM) is designed to simultaneously model the inner relationships between RS images and texts and enhance the visual features. IIEM consists of three feature interacting-enhancing (FIE) blocks, each of which contains an inter-modality relationship interacting (IMRI) sub-block and a visual feature enhancing (VFE) sub-block. The duty of IMRI is to exploit the hidden relations between cross-modal data, while the responsibility of VFE is to improve the visual features. By combining them, semantic bias can be mitigated, and the complex contents of RS images can be studied. Finally, the retrieval module (RM) is constructed to generate the matching scores for deciding the search results. Extensive experiments are conducted on four public RS data sets. The positive results demonstrate that our IEFT can achieve superior retrieval performance compared with many existing methods. Our source codes are available at https://github.com/TangXu-Group/Cross-modal-remote-sensing-image-and-text-retrieval-models/tree/main/IEFT.

Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations

Learning effective joint embedding for cross-modal data has always been a focus in the field of multimodal machine learning. We argue that during multimodal fusion, the generated multimodal embedding may be redundant, and the discriminative unimodal information may be ignored, which often interferes with accurate prediction and leads to a higher risk of overfitting. Moreover, unimodal representations also contain noisy information that negatively influences the learning of cross-modal dynamics. To this end, we introduce the multimodal information bottleneck (MIB), aiming to learn a powerful and sufficient multimodal representation that is free of redundancy and to filter out noisy information in unimodal representations. Specifically, inheriting from the general information bottleneck (IB), MIB aims to learn the minimal sufficient representation for a given task by maximizing the mutual information between the representation and the target and simultaneously constraining the mutual information between the representation and the input data. Different from general IB, our MIB regularizes both the multimodal and unimodal representations, which is a comprehensive and flexible framework that is compatible with any fusion methods. We develop three MIB variants, namely, early-fusion MIB, late-fusion MIB, and complete MIB, to focus on different perspectives of information constraints. Experimental results suggest that the proposed method reaches state-of-the-art performance on the tasks of multimodal sentiment analysis and multimodal emotion recognition across three widely used datasets. The codes are available at \url{https://github.com/TmacMai/Multimodal-Information-Bottleneck}.

Cross-Modal Data Augmentation for Tasks of Different Modalities

Data augmentation has become one of the keys to alleviating the over-fitting of models on training data and improving the generalization capabilities on testing data. Most existing data augmentation methods only focus on one modality, which is incapable when facing multiple data modalities. Some prior works try to interpolate with random coefficients in the latent space to generate new samples, which can generically work for any data modality. However, these works ignore the extra information conveyed by multimodality data. In fact, the extra information in one modality can provide semantic directions to generate more meaningful samples in another modality. This paper proposes Cross-modal Data Augmentation (CMDA), a simple yet effective data augmentation method to alleviate the over-fitting issue and improve the generalization performance. We evaluate CMDA on unsupervised and supervised tasks of different modalities, on which CMDA consistently and significantly outperforms baselines. For instance, CMDA improves the unsupervised anomaly detection baseline in vision modality from the AUROC <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$76.46\%, 73.07\%$</tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$64.36\%$</tex-math></inline-formula> to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$83.25\%, 76.22\%$</tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$70.57\%$</tex-math></inline-formula> on three different datasets, respectively. Besides, extensive experiments demonstrate that CMDA is applicable to various neural network architectures. Furthermore, prior methods that interpolate in the latent space need to work with downstream tasks to construct the latent space. In contrast, CMDA can work with or without downstream tasks, which makes the applicability of CMDA more extensive. Our source code is publicly available for non-commercial or research use at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Anfeather/CMDA</uri> .

RGB-D Crowd Counting With Cross-Modal Cycle-Attention Fusion and Fine-Coarse Supervision

To tackle the negative effect of the arbitrary crowd distribution on the counting task, in this article, we propose a novel RGB-D crowd counting approach, including a cross-modal cycle-attention fusion (CmCaF) model and a novel fine-coarse (FC) supervision. In the feature level, the CmCaF model combines the RGB feature and depth feature in a cycle-attention way so as to model the crowd distribution effectively. In the supervision level, the novel design of FC supervision could optimize the counting model from both the fine pixel-aware level and coarse region-aware level to enhance its sensitivity to the whole crowd distribution and the instance location. Extensive evaluations on benchmarks well illustrate the feasibility of the proposed approach for the RGB-D crowd counting, as well as RGB and RGB-T counting. And the ablation study demonstrates the effectiveness of its main components on both the feature representation of cross-modal data and the accurate estimation of the crowd distribution.

Infrared time-sensitive target detection technology based on cross-modal data augmentation

Infrared time-sensitive target detection technology based on cross-modal data augmentation

Deep Cross-Modal Hashing Based on Semantic Consistent Ranking

The amount of multi-modal data available on the Internet is enormous. Cross-modal hash retrieval maps heterogeneous cross-modal data into a single Hamming space to offer fast and flexible retrieval services. However, existing cross-modal methods mainly rely on the feature-level similarity between multi-modal data and ignore the relationship between relative rankings and label-level fine-grained similarity of neighboring instances. To overcome these issues, we propose a novel <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</u> eep <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C</u> ross-modal <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">H</u> ashing based on <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S</u> emantic <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C</u> onsistent <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R</u> anking (DCH-SCR) that comprehensively investigates the intra-modal semantic similarity relationship. Firstly, to the best of our knowledge, it is an early attempt to preserve semantic similarity for cross-modal hashing retrieval by combining label-level and feature-level information. Secondly, the inherent gap between modalities is narrowed by developing a ranking alignment loss function. Thirdly, the compact and efficient hash codes are optimized based on the common semantic space. Finally, we use the gradient to specify the optimization direction and introduce the Normalized Discounted Cumulative Gain (NDCG) to achieve varying optimization strengths for data pairs with different similarities. Extensive experiments on three real-world image-text retrieval datasets demonstrate the superiority of DCH-SCR over several state-of-the-art cross-modal retrieval methods.

Deep continual hashing with gradient-aware memory for cross-modal retrieval

Cross-modal hashing (CMH) has become widely used for large-scale multimedia retrieval. However, most current CMH methods focus on the closed retrieval scenario, not the real-world environments, i.e., complex and changing semantics. When data containing new class objects emerge, the current CMH has to retrain the model on all history training data, not the new data, to accommodate new semantics, but the never-stop upload of data on the Internet makes this impractical. In this paper, we devise a deep hashing method called Continual Cross-Modal Hashing with Gradient Aware Memory (CCMH-GAM) for learning binary codes of multi-label cross-modal data with increasing categories. CCMH-GAM is a two-step hashing architecture, one hashing network learns to hash the increasing semantics of data, i.e., label, into the semantic codes, and other modality-specific hashing networks learn to map data into the corresponding semantic codes. Specifically, to keep the encoding ability for old semantics, a regularization based on accumulating low-storage label-code pairs is designed for the former network. For the modality-specific networks, we propose a memory construction method via approximating the full episodic gradients of all data by some exemplars and derive its fast implementation with the upper bound of approximation error. Based on this memory, we propose a gradient projection method to theoretically improve the probability of old data’s code being unchanged after updating the model. Extensive experiments on three datasets demonstrate that CCMH-GAM can continually learn hash functions and yield state-of-the-art retrieval performance.

A dual tri-path CNN system for brain tumor segmentation

The research on developing CNN-based fully-automated brain-tumor-segmentation systems has been progressing rapidly. For the systems to be applicable in practice, a good processing quality and reliability are necessary. Moreover, as the parameters in a CNN are determined by training, based on statistical losses in training epochs, more parameters may cause more randomness in the process and a minimization of the number of parameters is required to achieve a good reproducibility of the results. To this end, the CNN in the proposed system has a unique structure with 2 distinguished characters. Firstly, the three paths of its feature extraction block are designed to extract, from the multi-modality input, comprehensive feature information of mono-modality, paired-modality and cross-modality data, respectively. Also, it has a particular three-branch classification block to distinguish pixels in each of the 3 intra-tumoral classes from the background. Each branch is trained separately so that the parameters are updated specifically with the corresponding ground truth data of target tumor areas. The convolutional layers of the system are custom-designed with specific purposes, resulting in a very simple config of 61,843 parameters in total. The proposed system has been tested extensively with BraTS2019 and BraTS2018 datasets. The mean Dice scores, obtained from the ten experiments on BraTS2019 validation samples, are 0.751 ± 0.007, 0.885 ± 0.002, 0.776 ± 0.004, for enhancing tumor, whole tumor and tumor core, respectively. The test results demonstrate that the proposed system is able to reproduce a high-quality segmentation result quite consistently. Furthermore, its extremely low computation complexity will facilitate its implementation/application in various environments.

Hybrid-attention based Feature-reconstructive Adversarial Hashing Networks for Cross-modal Retrieval

With the massive growth of data of various modal types, people no longer use a single modal retrieval method, but a cross-modal retrieval method when performing retrieval tasks. Such methods often need to store data efficiently while maintaining the characteristics of fast query. Because the hashing learning method can represent the original high-dimensional data through a simple and compact binary hash code, which can greatly compress the data size and facilitate data storage and mutual retrieval, the cross-modal hashing retrieval has gradually become a hot topic in recent years. However, how to bridge the gap between modalities to improve the retrieval performance further is still a challenging problem. In order to solve this problem, we propose a Hybrid-attention based Feature-reconstructive Adversarial Hashing (HFAH) networks for cross-modal retrieval. First, a label semantic guidance module is introduced to guide the extraction process of text features and image features through the learning of labels, so as to fully maintain the semantic similarity between different modal data. Then, the hybrid-attention module is introduced to make the extracted data contain richer semantic information. Finally, the feature reconstruction network is used to make the relevant degree between similar cross-modal data pairs higher than that between dissimilar data pairs. Related experiments on two benchmark datasets confirm to us that HFAH performs better than several existing cross- modal retrieval methods.