Multimodal Remote Sensing Image Research Articles

LDS2AE: Local Diffusion Shared-Specific Autoencoder for Multimodal Remote Sensing Image Classification with Arbitrary Missing Modalities

Recent research on the joint classification of multimodal remote sensing data has achieved great success. However, due to the limitations imposed by imaging conditions, the case of missing modalities often occurs in practice. Most previous researchers regard the classification in case of different missing modalities as independent tasks. They train a specific classification model for each fixed missing modality by extracting multimodal joint representation, which cannot handle the classification of arbitrary (including multiple and random) missing modalities. In this work, we propose a local diffusion shared-specific autoencoder (LDS2AE), which solves the classification of arbitrary missing modalities with a single model. The LDS2AE captures the data distribution of different modalities to learn multimodal shared feature for classification by designing a novel local diffusion autoencoder which consists of a modality-shared encoder and several modality-specific decoders. The modality-shared encoder is designed to extract multimodal shared feature by employing the same parameters to map multimodal data into a shared subspace. The modality-specific decoders put the multimodal shared feature to reconstruct the image of each modality, which facilitates the shared feature to learn unique information of different modalities. In addition, we incorporate masked training to the diffusion autoencoder to achieve local diffusion, which significantly reduces the training cost of model. The approach is tested on widely-used multimodal remote sensing datasets, demonstrating the effectiveness of the proposed LDS2AE in addressing the classification of arbitrary missing modalities. The code is available at https://github.com/Jiahuiqu/LDS2AE.

Robust registration of multi-modal remote sensing images based on multi-dimensional oriented self-similarity features

Registration of multi-modal remote sensing images (MRSI) is crucial for unlocking the full potential of heterogeneous remote sensing imagery. However, achieving accurate registration among MRSI is challenging due to the trade-off between geometric invariance and matching accuracy, caused by differences in signal-to-noise ratio and nonlinear radiometric distortion (NRD) arising from varying imaging mechanisms. To tackle the challenge, this paper proposes a lightweight and hybrid feature-guided registration algorithm for MRSI called the hybrid registration algorithm based on multi-dimensional oriented self-similarity features (MOSS). MOSS leverages the advantages of multi-dimensional oriented self-similarity features to progressively enhance registration performance. In the hybrid feature coarse matching stage, oriented self-similarity features are extracted from MRSI, and their directional information is utilized for feature description to estimate the initial affine transformation. The fine matching under multi-dimensional oriented self-similarity features stage takes the outputs of the coarse matching stage to perform a template-like matching process. To evaluate the performance of MOSS, comprehensive experiments are conducted using six different combinations of MRSI, and seven state-of-the-art registration algorithms are selected for comparison. The experimental results demonstrate that MOSS outperforms the compared methods, with the number of correct matches being at least about 1.6 times higher than the comparison methods. Moreover, MOSS exhibits the lowest root mean square error across all experiments, with an average RMSE of 1.86 pixels, achieving an RMSE within 2 pixels. This highlights its effectiveness in achieving precise alignment and robust registration of MRSI.

Uncertainty-Aware Contrastive Learning for Semi-Supervised Classification of Multimodal Remote Sensing Images

Uncertainty-Aware Contrastive Learning for Semi-Supervised Classification of Multimodal Remote Sensing Images

Robust Multimodal Remote Sensing Image Matching Based on Enhanced Oriented Self-Similarity Descriptor

Robust Multimodal Remote Sensing Image Matching Based on Enhanced Oriented Self-Similarity Descriptor

Triplet Contrastive Learning Framework With Adversarial Hard-Negative Sample Generation for Multimodal Remote Sensing Images

Triplet Contrastive Learning Framework With Adversarial Hard-Negative Sample Generation for Multimodal Remote Sensing Images

Multimodal Remote Sensing Image Matching via Learning Features and Attention Mechanism

Multimodal Remote Sensing Image Matching via Learning Features and Attention Mechanism

Extended Cumulative Structural Feature Matching Method for Multimodal Remote Sensing Images

Extended Cumulative Structural Feature Matching Method for Multimodal Remote Sensing Images

Fine-Grained Ship Recognition Based on Visible and Near-Infrared Multimodal Remote Sensing Images: Dataset, Methodology and Evaluation

Fine-Grained Ship Recognition Based on Visible and Near-Infrared Multimodal Remote Sensing Images: Dataset, Methodology and Evaluation

Multiloss Adversarial Attacks for Multimodal Remote Sensing Image Classification

Multiloss Adversarial Attacks for Multimodal Remote Sensing Image Classification

Multimodal Remote Sensing Image Registration Based on Adaptive Spectrum Congruency

Multimodal Remote Sensing Image Registration Based on Adaptive Spectrum Congruency

Multimodal Remote Sensing Image Matching Based on Weighted Structure Saliency Feature

Multimodal Remote Sensing Image Matching Based on Weighted Structure Saliency Feature

Dual Graph Learning Affinity Propagation for Multimodal Remote Sensing Image Clustering

Dual Graph Learning Affinity Propagation for Multimodal Remote Sensing Image Clustering

Classification of multi-modal remote sensing images based on knowledge graph

ABSTRACT With the development of remote sensing (RS) technology, single-modal data alone has gradually become difficult to meet the requirement for high accuracy of RS image classification. As a result, multi-modal RS image classification has become a hot research topic. However, multi-modal RS image classification faces challenges in effectively utilizing advanced semantic information regarding the relationships between land cover classes and extracting discriminative features from the data. Based on this, a multi-modal RS image classification method based on knowledge graph (KG) is proposed. Firstly, graph topology is used to align the hyperspectral image (HSI) and multispectral image (MSI) features extracted by graph convolution networks. Constraint alignment is performed on different feature graphs to reduce the difficulty of fusion and the false recognition rate. Then, we use self-attention and cross-attention to purposefully fuse HSI and MSI to obtain discriminative features rich in two modal information and achieve feature weighted fusion. Finally, a KG based on object spatial relationships is constructed to obtain spatial relationships between different classes to assist in multi-modal RS image classification. The experimental results on the Houston and Ausburg datasets demonstrate that the proposed method achieves overall accuracy of 90.40% and 90.85%, respectively, both of which are more than 3% higher than existing classification methods. The results indicate that our method has better classification performance and can provide a useful reference for RS image classification research.

SRTPN: Scale and Rotation Transform Prediction Net for Multimodal Remote Sensing Image Registration

How to recover geometric transformations is one of the most challenging issues in image registration. To alleviate the effect of large geometric distortion in multimodal remote sensing image registration, a scale and rotate transform prediction net is proposed in this paper. First, to reduce the scale between the reference and sensed images, the image scale regression module is constructed via CNN feature extraction and FFT correlation, and the scale of sensed image can be recovered roughly. Second, the rotation estimate module is developed for predicting the rotation angles between the reference and the scale-recovered images. Finally, to obtain the accurate registration results, LoFTR is employed to match the geometric-recovered images. The proposed registration network was evaluated on GoogleEarth, HRMS, VIS-NIR and UAV datasets with contrast differences and geometric distortions. The experimental results show that the number of correct matches of our model reached 74.6%, and the RMSE of the registration results achieved 1.236, which is superior to the related methods.

Advances and Challenges in Multimodal Remote Sensing Image Registration

Over the past few decades, with the rapid development of global aerospace and aerial remote sensing technology, the types of sensors have evolved from the traditional monomodal sensors (e.g., optical sensors) to the new generation of multimodal sensors e.g., multispectral, hyperspectral, light detection and ranging (LiDAR) and synthetic aperture radar (SAR) sensors. These advanced devices can dynamically provide various and abundant multimodal remote sensing images with different spatial, temporal, and spectral resolutions according to different application requirements. Since then, it is of great scientific significance to carry out the research of multimodal remote sensing image registration, which is a crucial step for integrating the complementary information among multimodal data and making comprehensive observations and analysis of the Earths surface. In this work, we will present our own contributions to the field of multimodal image registration, summarize the advantages and limitations of existing multimodal image registration methods, and then discuss the remaining challenges and make a forward-looking prospect for the future development of the field.

MoCG: Modality Characteristics-Guided Semantic Segmentation in Multimodal Remote Sensing Images

MoCG: Modality Characteristics-Guided Semantic Segmentation in Multimodal Remote Sensing Images

Combining Phase Congruency and Self-Similarity Features for Multimodal Remote Sensing Image Matching

The structural features using self-similarity have become more popular for multimodal remote sensing image matching. However, mostly because of significant geometric distortions and nonlinear intensity differences between images, these methods produce a limited matching performance when directly applied to multimodal remote sensing images. To address that, we propose a novel feature descriptor named pyramid features of orientated self-similarity (POSS) for multimodal remote sensing image matching, which integrates phase congruency (PC) into the self-similarity model for better encoding structural information. Unlike these conventional self-similarity-based descriptors, the POSS is constructed by using PC instead of image intensity in a pyramid manner and thus has better discriminability and robustness to modality variations. In addition, we design a uniform multimoment of PC (UMMPC) detector for feature detection, which can improve the number, distribution, and repeatability of feature points. Experimental results demonstrate its superior applicability of the proposed method over the state-of-the-art methods, showing much better matching performance.

RingMo-SAM: A Foundation Model for Segment Anything in Multimodal Remote-Sensing Images

RingMo-SAM: A Foundation Model for Segment Anything in Multimodal Remote-Sensing Images

Modality-Invariant Structural Feature Representation for Multimodal Remote Sensing Image Matching

Modality-Invariant Structural Feature Representation for Multimodal Remote Sensing Image Matching

Unsupervised Multimodal Remote Sensing Image Registration via Domain Adaptation

Unsupervised Multimodal Remote Sensing Image Registration via Domain Adaptation