Remote Sensing Scene Research Articles

Relation-Attention Networks for Remote Sensing Scene Classification

Remote sensing (RS) scene classification plays an important role in a wide range of RS applications. Recently, convolutional neural networks (CNNs) have been applied to the field of scene classification in RS images and achieved impressive performance. However, to classify RS scenes, most of the existing CNN methods either utilize the high-level features from the last convolutional layer of CNNs, missing much important information existing at the other levels, or directly fuse the features at different levels, bringing redundant and/or mutually exclusive information. Inspired by the attention mechanism of the human visual system, in this article, we explore a novel relation-attention model and design an end-to-end relation-attention network (RANet) to learn powerful feature representations of multiple levels to further improve the classification performance. First, we propose to extract convolutional features at different levels by pretrained CNNs. Second, a multiscale feature computation module is constructed to connect features at different levels and generate multiscale semantic features. Third, a novel relation-attention model is designed to focus on the critical features whilst avoiding the use of redundant and even distractive ones by exploiting the scale contextual information. Finally, the resulting relation-attention features are concatenated and fed into a softmax layer for the final classification. Experiments on four well-known RS scene classification datasets (UC-Merced, WHU-RS19, AID, and OPTIMAL-31) show that our method outperforms some state-of-the-art algorithms.

Historical Information-Guided Class-Incremental Semantic Segmentation in Remote Sensing Images

Despite the extraordinary success of the deep architectures on semantic segmentation for remote sensing (RS) images, they have difficulties in learning new classes from a sequential data stream because of catastrophic forgetting. Continual learning for semantic segmentation (CSS) is an emerging trend for its capability to cope with the above problems effectively. However, old classes from previous steps are collapsed into the background, which further aggravates the challenge of CSS in the RS scene. In this article, we revisit the knowledge distillation (KD) strategy and the characteristics of class-incremental semantic segmentation (CISS) and then present a generalized and effective framework to learn new classes while preserving knowledge of the learned classes. In particular, we propose two novel historical information-guided modules: the feature global perception module and the label reconstruction (LR) module. The former enables the current model to pay more attention to the region related to the old categories identified by the historical information when learning new classes. Meanwhile, the latter retrieves pixels belonging to the learned classes from the background to handle the background shift problem and maintain the high performance of old classes. We have conducted comprehensive experiments on two RS semantic segmentation datasets of Instance Segmentation in Aerial Images Dataset (iSAID) and Gao Fen (GF) challenge semantic segmentation dataset (GCSS). The experimental results outperform the current state-of-the-art methods in most incremental settings, which demonstrates the effectiveness of the proposed framework.

Contextual Information-Preserved Architecture Learning for Remote-Sensing Scene Classification

Convolutional neural networks (CNNs) have recently been widely used in remote-sensing scene classification. Additionally, it is becoming very popular to automatically learn specific CNN architectures for specific data sets. The rich contextual information in high-resolution remote-sensing images (RSIs) is critical to remote-sensing intelligent understanding tasks. However, architecture learning approaches tend to simplify the original data (i.e., resizing images to smaller resolution) for efficiency, yet result in contextual information loss of RSIs. In this article, we proposed a contextual information-preserved architecture learning (CIPAL) framework for remote-sensing scene classification to utilize the contextual information in RSIs as much as possible during the architecture learning process. We introduce channel compression into CIPAL, which can reduce the memory and time consumption of architecture learning and make it possible to construct a larger architecture space. We add potential operators that are rarely used for scene classification tasks (i.e., atrous convolution) into the architecture space to explore unknown architectures that are more suitable for remote-sensing scenes. The experimental results on four remote-sensing scene classification benchmarks indicate that CIPAL learns architectures with less time consumption than similar works, and the newly found architectures outperform popular hand-designed architectures for better use of contextual information in RSIs. Different architectures are good at learning different representations, and our proposed architecture learning method potentially helps us understand which types of representations are crucial for RSI intelligent understanding.

Transferring CNN With Adaptive Learning for Remote Sensing Scene Classification

Accurate classification of remote sensing (RS) images is perennial topic of interest in the RS community. Recently, transfer learning, especially for fine-tuning pre-trained convolutional neural networks (CNNs), has been proposed as a feasible strategy for RS scene classification. However, because the target domain (i.e., the RS images) and the source domain (e.g., ImageNet) are quite different, simply using the model pre-trained on an ImageNet dataset presents some difficulties. The RS images and the pre-trained models need to be properly adjusted to build a better classification system. In this study, an adaptive learning strategy for transferring a CNN-based model is proposed. First, an adaptive transform is used to adjust the original size of the RS image to a certain size, which is tailored to the input of the subsequent pre-trained model. Then, an adaptive transferring model is proposed to automatically learn what knowledge from the pre-trained model should be transferred to the RS scene classification model. Finally, in combination with a label smoothing approach, adaptive label is presented to generate soft labels based on the statistics of the classification model predictions for each category, which is beneficial for learning the relationships between the target and non-target categories of scenes. In general, the proposed methods adaptively manage the input, model, and label simultaneously, which leads to better classification performance for RS scene classification. The proposed methods are tested on three widely-used data sets and the obtained results show that the proposed methods provide competitive classification accuracy compared to the state-of-the-art methods.

Multi-scale Convolutional Neural Network Driven by Sparse Second-order Attention Mechanism for Remote Sensing Scene Classification

Multi-scale Convolutional Neural Network Driven by Sparse Second-order Attention Mechanism for Remote Sensing Scene Classification

Multi-Representation Dynamic Adaptation Network for Remote Sensing Scene Classification

In recent years, convolutional neural networks (CNNs) have made significant progress in remote sensing scene classification (RSSC) tasks. Because obtaining a large number of labeled images is time-consuming and expensive and the generalization ability of supervised models is limited, domain adaptation is widely introduced into RSSC. However, existing adaptation approaches mainly aim to align the distribution of features in a single representation space, which results in losing information and limiting the spatial range for extracting domain-invariant features. In addition, some of the methods simultaneously align pixel-level (local) and image-level (global) features for better results but suffer from searching for the best weight of the two parts manually, which is time-consuming and computing-expensive. To overcome the above issues, a novel feature fusion-and-alignment approach named Multi-Representation Dynamic Adaption Network (MRDAN) is proposed for cross-domain RSSC. Concretely, a Feature-Fusion Adaptation (FFA) module is embedded into the network, which maps samples to multiple representations and fuses them to obtain a broader domain-invariant feature space. Based on this hybrid space, we introduce a cross-domain Dynamic Feature-Alignment Mechanism (DFAM) to quantitatively evaluate and adjust the relative importance of the local and global adaptation losses during domain adaptation. The experimental results on the 12 transfer tasks between the UC Merced land-use, WHU-RS19, AID, and RSSCN7 data sets demonstrate the effectiveness of the proposed MRDAN over the state-of-the-art domain adaptation methods in RSSC.

NAS-Based CNN Channel Pruning for Remote Sensing Scene Classification

Recently, convolutional neural network (CNN)-based remote sensing scene classification has achieved great success. However, the prohibitively expensive computation and storage requirements of state-of-the-art models have hindered the deployment of CNNs on on- board platforms. In this letter, we propose a differentiable neural architecture search (NAS)-based channel pruning method to automatically prune the CNN models. In the proposed method, the importance of each output channel is measured by a trainable score. The scores are optimized by an NAS method to search a good-performance pruned structure. After the search process, a global score threshold is adopted to derive the pruned model. A cost-awareness loss is proposed for the search process to encourage the floating-point operation (FLOP) compression ratio of the pruned model coverage to a desired value. We apply the proposed method to ResNet-34 and VGG-16 to verify the performance. The NWPU-RESISC-45 and UC Merced Land-Use (UCM) datasets are used for the performance evaluation. A comparison with state-of-the-art pruning methods demonstrates that the proposed method can achieve competitive performance with a similar reduction in FLOP.

Searching for CNN Architectures for Remote Sensing Scene Classification

Convolutional neural network (CNN) models for remote sensing (RS) scene classification are largely built on pretrained networks that are trained on the general-purpose ImageNet dataset in computer vision. The pretrained networks can easily be adapted for transfer learning in RS scene classification. However, the accuracy of transfer learning may decline as RS images are considerably different from other images. Thus, the pretrained CNN model learned on ImageNet may not be sufficient for the accurate classification of RS image scenes. Furthermore, most of the pretrained models have large memory footprints, which place a further burden on computational requirements. In this work, we explore SLGE-based random search with early stopping in the search for CNN architectures for both single-label and multilabel RS scene classification tasks. In SLGE, the architecture search space is capable of representing multipath Inception-like modular cells with skip-connections similar to human-expert designs. The experimental results on four RS scene classification benchmarks show that the automatically discovered networks demonstrate the promising capability in classifying multispectral satellite image scenes compared with fine-tuned pretrained CNN models. Using fewer parameters with 0.56B FLOPS, our best network achieves a classification accuracy rate of 96.56% and 96.10% on NWPU-RESISC45 single-label and AID single-label RGB aerial image datasets, respectively, and the classification accuracy rate of 99.76% and 93.89% on EuroSAT single-label and BigEarthNet multilabel multispectral satellite image datasets, respectively. The results position our approach among the best of the state of the art.

Embedded Self-Distillation in Compact Multibranch Ensemble Network for Remote Sensing Scene Classification

Remote sensing (RS) image scene classification task faces many challenges due to the interference from different characteristics of different geographical elements. To solve this problem, we propose a multi-branch ensemble network to enhance the feature representation ability by fusing features in final output logits and intermediate feature maps. However, simply adding branches will increase the complexity of models and decline the inference efficiency. On this issue, we embed self-distillation (SD) method to transfer knowledge from ensemble network to main-branch in it. Through optimizing with SD, main-branch will have close performance as ensemble network. During inference, we can cut other branches to simplify the whole model. In this paper, we first design compact multi-branch ensemble network, which can be trained in an end-to-end manner. Then, we insert SD method on output logits and feature maps. Compared to previous methods, our proposed architecture (ESD-MBENet) performs strongly on classification accuracy with compact design. Extensive experiments are applied on three benchmark RS datasets AID, NWPU-RESISC45 and UC-Merced with three classic baseline models, VGG16, ResNet50 and DenseNet121. Results prove that our proposed ESD-MBENet can achieve better accuracy than previous state-of-the-art (SOTA) complex models. Moreover, abundant visualization analysis make our method more convincing and interpretable.

SCL-MLNet: Boosting Few-Shot Remote Sensing Scene Classification via Self-Supervised Contrastive Learning

Few-shot classification aims at recognizing novel categories from low data regimes based on prior knowledge. However, the existing methods for few-shot scene classification have limitations on using few annotated data and do not fully consider the intra-class samples with classification targets in different sizes, which lead to poor feature representation. To address these problems, this study introduces an end-to-end framework called self-supervised contrastive learning-based metric learning network (SCL-MLNet) for few-shot remote sensing (RS) scene classification. On one hand, we weave self-supervised contrastive learning into few-shot classification algorithms through multi-task learning, enabling feature extractors to learn representative image features from few annotated samples. Moreover, we devise a new loss function to train the proposed model end-to-end and speed up the convergence of the model. On the other hand, considering the differences between intra-class samples, we introduce a novel attention module embedded in the feature extractor to fuse multi-scale spatial features from the classification targets in different sizes. In our experiments, SCL-MLNet is evaluated on three public benchmark datasets. The results demonstrate that SCL-MLNet achieves state-of-the-art performance for few-shot remote sensing scene classification.

SGMNet: Scene Graph Matching Network for Few-Shot Remote Sensing Scene Classification

Few-Shot Remote Sensing Scene Classification (FSRSSC) is an important task, which aims to recognize novel scene classes with few examples. Recently, several studies attempt to address the FSRSSC problem by following few-shot natural image classification methods. These existing methods have made promising progress and achieved superior performance. However, they all overlook two unique characteristics of remote sensing images: (i) object co-occurrence that multiple objects tend to appear together in a scene image and (ii) object spatial correlation that these co-occurrence objects are distributed in the scene image following some spatial structure patterns. Such unique characteristics are very beneficial for FSRSSC, which can effectively alleviate the scarcity issue of labeled remote sensing images since they can provide more refined descriptions for each scene class. To fully exploit these characteristics, we propose a novel scene graph matching-based meta-learning framework for FSRSSC, called SGMNet. In this framework, a scene graph construction module is carefully designed to represent each test remote sensing image or each scene class as a scene graph, where the nodes reflect these co-occurrence objects meanwhile the edges capture the spatial correlations between these co-occurrence objects. Then, a scene graph matching module is further developed to evaluate the similarity score between each test remote sensing image and each scene class. Finally, based on the similarity scores, we perform the scene class prediction via a nearest neighbor classifier. We conduct extensive experiments on UCMerced LandUse, WHU19, AID, and NWPU-RESISC45 datasets. The experimental results show that our method obtains superior performance over the previous state-of-the-art methods.

Improving Few-Shot Remote Sensing Scene Classification With Class Name Semantics

Few-shot remote sensing scene classification (FSRSSC) has been used for new class recognition in the presence of a limited number of labeled samples. The representation vector (prototype) of categories obtained using images only confronts some challenges, such as insufficient generalization when the number of samples is too small. To address this problem, we propose a new FSRSSC method based on prototype networks, named CNSPN, which combines semantic information of class names (name of the scene categories, such as aircraft, harbor, and bridge). First, CNSPN extracts semantics for class names using a pre-trained word-embedding model, which enriches the feature representation ability of the category at the source. Then, an enhanced fusion prototype is generated by fusing the semantic information of text and visual information in the image through a multimodal prototype fusion module (MPFM). Finally, the query image is classified by measuring the distance between the query sample and the visual prototype, and between the query sample and the fusion prototype. Comparative experiments on the NWPU-RESISC45 and RSD46-WHU datasets show that the proposed method significantly improves FSRSSC performance. Code is available at https://github.com/RS-CSU/CNSPN.git.

Context Residual Attention Network for Remote Sensing Scene Classification

Remote sensing image scene classification has attracted much attention due to its wide application. In this paper, a new end-to-end attentive context network under the guidance of human visual system has been proposed. The network can focus on some critical regions of the image selectively and then extract high-level feature information so as to generalize the whole image. The contributions of this letter are as follows: firstly, a novel attention structure which expresses the attention of image by stacking attention modules is designed; secondly, the context information of the image is utilized to make a holistic analysis of the attention mechanism on the basis of the top-down feedforward structure. Experiments performed on several datasets demonstrate that the proposed framework can obtain outstanding performance compared with state-of-the-art approaches.

GPU-Friendly Neural Networks for Remote Sensing Scene Classification

Convolutional neural networks (CNNs) have proven to be very efficient for the analysis of remote sensing (RS) images. Due to the inherent complexity of extracting features from these images, along with the increasing amount of data to be processed (and the diversity of applications), there is a clear tendency to develop and employ increasingly deep and complex CNNs. In this regard, graphics processing units (GPUs) are frequently used to optimize their execution, both for the training and inference stages, optimizing the performance of neural models through their many-core architecture. Hence, the efficient use of the GPU resources should be at the core of optimizations. This letter analyzes the possibilities of using a new family of CNNs, denoted as TResNets, to provide an efficient solution to the RS scene classification problem. Moreover, the considered models have been combined with mixed precision to enhance their training performance. Our experimental results, conducted over three publicly available RS data sets, show that the proposed networks achieve better accuracy and more efficient use of GPU resources than other state-of-the-art networks. Source code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/mhaut/GPUfriendlyRS</uri> .

Multilayer Feature Fusion Network With Spatial Attention and Gated Mechanism for Remote Sensing Scene Classification

Remote sensing (RS) scene classification has attracted extensive attention due to its large number of applications. Recently, convolutional neural networks (CNNs) methods have shown impressive ability of feature learning in RS scene classification. However, the performance is still limited by large-scale variance and complex background. To address these problems, we present a multilayer feature fusion network with spatial attention and gated mechanism (MLF2Net_SAGM) for RS scene classification. At first, the backbone is employed to extract multilayer convolutional features. Then, a residual spatial attention module (RSAM) is proposed to enhance discriminative regions of the multilayer feature maps, and key areas can be harvested. Finally, the multilayer spatial calibration features are fused to form the final feature map, and a gated fusion module (GFM) is designed to eliminate feature redundancy and mutual exclusion (FRME). To verify the effectiveness of the proposed method, we conduct comparative experiments based on three widely used RS image scene classification benchmarks. The results show that the direct fusion of multilayer features via element-wise addition leads to FRME, whereas our method fuses multilayer features more effectively and improves the performance of scene classification.

Focal Cosine Metric and Adaptive Attention Module for Remote Sensing Scene Classification With Siamese Convolutional Neural Networks

Convolutional neural networks (CNNs) have been widely used in remote sensing (RS) scene classification tasks due to their remarkable feature representation and inference capability. The complexity of RS images not only brings the challenges of high inter-class similarity and large intra-class diversity, but also introduces the problem that category-relevant regions are insufficiently prominent in feature extraction. Siamese CNNs with feature similarity measurement are chosen in some applications to overcome the former issue, but most ignore the randomness of input sample pairs. This makes the Siamese CNNs not focus enough on challenging samples, which limits the training efficiency. We propose the focal cosine metric (FCM) block that combines the cosine similarity metric and the threshold control to achieve sample selection, thereby completing network learning more efficiently. FCM only permits the misclassified focal samples to participate in similarity measurement based on Siamese CNN. It flexibly mitigates the misclassification caused by the high inter-class similarity and large intra-class diversity. Moreover, the adaptive attention (AA) module is designed to stress the pivotal target regions and assist in the similarity measurement of Siamese CNN. This is realized by adaptively assigning high weights to key targets with learnable guided vectors. It enables the model to focus on the details of intra-class similarities or inter-class differences in sample pairs, and thus reduces the difficulty of model optimization. Encouraging experimental results on three public data sets demonstrate the effectiveness of the novel Siamese CNN-based method with FCM and AA and show its superiority compared to other state-of-the-art scene classification methods.

Cosine Margin Prototypical Networks for Remote Sensing Scene Classification

In practical applications, remote-sensing scene classification tasks generally exhibit data shift problems. In this situation, images have large data discrepancies, leading to diffused features and performance degradation. To address the data shift problem, we propose cosine margin prototypical networks. Specifically, we adopt a cosine margin to constrain strictly features, generating well-clustered and discriminative features. With the cosine margin, our method can alleviate data discrepancies by obtaining discriminative features and further addresses the data shift problem well. We conduct extensive experiments on various datasets and achieve 0.03%–7.11% higher accuracy than existing methods. Competitive experimental results demonstrate that our method can solve the data shift problems well in remote-sensing scene classification.

Dropout-Based Adversarial Training Networks for Remote Sensing Scene Classification

Scene classification in remote sensing (RS) images is a challenging task due to the lack of well-labeled data. Recently, deep transfer learning (DTL) has been proposed to handle this task. However, the intraclass variations and interclass similarities remain challenges. To handle these challenges, this letter presents a novel dropout-based adversarial training network (DATN) for RS scene classification. Specifically, a dropout-based label classifier (DLC) module is designed to reduce the selection of ambiguous features on class boundaries. Then, a dropout-based domain discriminator (DDD) module is constructed to capture multimodal structures of RS images so as to achieve fine-grained alignment between cross-domain distributions. Third, a joint distribution of features and labels is built to further enhance the performance. Experiments on seven public RS datasets show that our model outperforms several states of the art (SOTAs) under different conditions. The code of our method is publicly available at <uri>https://github.com/WangXin81/DATN-Submitted-to-IEEE-GRSL</uri>.

Explaining the Effects of Clouds on Remote Sensing Scene Classification

Most of Earth is covered by haze or clouds, impeding the constant monitoring of our planet. Preceding works have documented the detrimental effects of cloud coverage on remote sensing applications and proposed ways to approach this issue. However, up-to-now, little effort has been spent on understanding how exactly atmospheric disturbances impede the application of modern machine learning methods to Earth observation data. Specifically, we consider the effects of haze and cloud coverage on a scene classification task. We provide a thorough investigation of how classifiers trained on cloud-free data fail once they encounter noisy imagery – a common scenario encountered when deploying pretrained models for remote sensing to real use cases. We show how and why remote sensing scene classification suffers from cloud coverage. Based on a multi-stage analysis, including explainability approaches applied to the predictions, we work out four different types of effects that clouds have on the scene prediction. The contribution of our work is to deepen the understanding of the effects of clouds on common remote sensing applications, and consequently guide the development of more robust methods.

Rethinking Few-Shot Remote Sensing Scene Classification: A Good Embedding Is All You Need?

In recent years, few-shot remote sensing scene classification (FSRSSC) has attracted more and more attention. For FSRSSC, most methods currently focus on designing a meta-learning algorithm, which obtains meta-knowledge from limited samples and then applies it to novel tasks. In this work, on the one hand, we optimize the training pipeline of the feature extractor; on the other hand, we apply a novel model fusion method further to optimize the feature extractor capability of the feature extractor. We show a novel few-shot remote sensing scene classification baseline: learning two feature representations through using two self-supervised methods on the meta-training set and then fusing the two representations into one. Then, training a linear classifier on this representation achieves state-of-the-art performance. It shows that training a good feature extractor can be more efficient than complex meta-learning algorithms for FSRSSC. We believe that our results can inspire a rethinking of few-shot remote sensing scene classification benchmarks.