Remote Sensing Scene Classification Research Articles

Self-supervised embedding for generalized zero-shot learning in remote sensing scene classification

Self-supervised embedding for generalized zero-shot learning in remote sensing scene classification

Faster and Better: A Lightweight Transformer Network for Remote Sensing Scene Classification

Remote sensing (RS) scene classification has received considerable attention due to its wide applications in the RS community. Many methods based on convolutional neural networks (CNNs) have been proposed to classify complex RS scenes, but they cannot fully capture the context in RS images because of the lack of long-range dependencies (the dependency relationship between two distant elements). Recently, some researchers fine-tuned the large pretrained vision transformer (ViT) on small RS datasets to extract long-range dependencies effectively in RS scenes. However, it usually takes more time to fine-tune the ViT on account of high computational complexity. The lack of good local feature representation in the ViT limits classification performance improvement. To this end, we propose a lightweight transformer network (LTNet) for RS scene classification. First, a multi-level group convolution (MLGC) module is presented. It enriches the diversity of local features and requires a lower computational cost by co-representing multi-level and multi-group features in a single module. Then, based on the MLGC module, a lightweight transformer block, LightFormer, was designed to capture global dependencies with fewer computing resources. Finally, the LTNet was built using the MLGC and LightFormer. The experiments of fine-tuning the LTNet on four RS scene classification datasets demonstrate that the proposed network achieves a competitive classification performance under less training time.

Real-time scene classification of unmanned aerial vehicles remote sensing image based on Modified GhostNet.

Unmanned Aerial Vehicles (UAVs) play an important role in remote sensing image classification because they are capable of autonomously monitoring specific areas and analyzing images. The embedded platform and deep learning are used to classify UAV images in real-time. However, given the limited memory and computational resources, deploying deep learning networks on embedded devices and real-time analysis of ground scenes still has challenges in actual applications. To balance computational cost and classification accuracy, a novel lightweight network based on the original GhostNet is presented. The computational cost of this network is reduced by changing the number of convolutional layers. Meanwhile, the fully connected layer at the end is replaced with the fully convolutional layer. To evaluate the performance of the Modified GhostNet in remote sensing scene classification, experiments are performed on three public datasets: UCMerced, AID, and NWPU-RESISC. Compared with the basic GhostNet, the Floating Point Operations (FLOPs) are reduced from 7.85 MFLOPs to 2.58 MFLOPs, the memory is reduced from 16.40 MB to 5.70 MB, and the predicted time is improved by 18.86%. Our modified GhostNet also increases the average accuracy (Acc) (4.70% in AID experiments, 3.39% in UCMerced experiments). These results indicate that our Modified GhostNet can improve the performance of lightweight networks for scene classification and effectively enable real-time monitoring of ground scenes.

Ebola optimization with modified DarkNet‐53 model for scene classification and security on Internet of Things in smart cities

Remote sensing (RS) has developed significantly with the progress of the Internet of Things (IoT) which is allowed the cheap and fast acquisition of data in millions and billions of interrelated devices utilized throughout the whole world. RS scene classifier that purposes for classifying scene types for RS images has wide applications in several domains like urban planning, national defence security, environmental monitoring, and natural hazard detection. State-of-the-art deep learning (DL) successes are performed in a novel wave of RS scene classification applications, but it is the absence of explainability and trustworthiness. An intrusion detection system (IDS) plays a vital role to ensure security in the RS-based IoT environment. In this aspect, this study presents an ebola optimization algorithm with deep learning-based scene classification and intrusion detection (EOADL-SCID) technique on IoT-enabled remote sensing images. The aim of the EOADL-SCID system lies in the effectual scene classification of remote sensing images and intrusion detection. It involves a two-stage procedure. In the initial stage, the EOADL-SCID algorithm involves a modified DarkNet-53 feature extractor, EOA-based hyperparameter tuning, and graph convolution network (GCN) based classification. Next, in the second stage, the intrusion detection process takes place via two subprocesses namely variational autoencoder (VAE) based intrusion detection and skill optimization algorithm (SOA) based parameter tuning. The simulation outcomes of the EOADL-SCID approach are tested utilizing two benchmark databases and the experimental outcomes highlighted the improved performance of the EOADL-SCID algorithm on scene classification and intrusion classification processes.

A Lightweight Dual-Branch Swin Transformer for Remote Sensing Scene Classification

The main challenge of scene classification is to understand the semantic context information of high-resolution remote sensing images. Although vision transformer (ViT)-based methods have been explored to boost the long-range dependencies of high-resolution remote sensing images, the connectivity between neighboring windows is still limited. Meanwhile, ViT-based methods commonly contain a large number of parameters, resulting in a huge computational consumption. In this paper, a novel lightweight dual-branch swin transformer (LDBST) method for remote sensing scene classification is proposed, and the discriminative ability of scene features is increased through combining a ViT branch and convolutional neural network (CNN) branch. First, based on the hierarchical swin transformer model, LDBST divides the input features of each stage into two parts, which are then separately fed into the two branches. For the ViT branch, a dual multilayer perceptron structure with a depthwise convolutional layer, termed Conv-MLP, is integrated into the branch to boost the connections with neighboring windows. Then, a simple-structured CNN branch with maximum pooling preserves the strong features of the scene feature map. Specifically, the CNN branch lightens the LDBST, by avoiding complex multi-head attention and multilayer perceptron computations. To obtain better feature representation, LDBST was pretrained on the large-scale remote scene classification images of the MLRSN and RSD46-WHU datasets. These two pretrained weights were fine-tuned on target scene classification datasets. The experimental results showed that the proposed LDBST method was more effective than some other advanced remote sensing scene classification methods.

A lightweight and stochastic depth residual attention network for remote sensing scene classification

AbstractDue to the rapid development of satellite technology, high‐spatial‐resolution remote sensing (HRRS) images have highly complex spatial distributions and multiscale features, making the classification of such images a challenging task. The key to scene classification is to accurately understand the main semantic information contained in images. Convolutional neural networks (CNNs) have outstanding advantages in this field. Deep CNNs (D‐CNNs) with better performance tend to have more parameters and higher complexity. However, shallow CNNs have difficulty extracting the key features of complex remote sensing images. In this paper, we propose a lightweight network with a random depth strategy for remote sensing scene classification (LRSCM). We construct a convolutional feature extraction module, DCAB, which incorporates depthwise separable convolutional and inverted residual structures, effectively reducing the numbers of required parameters and computations, and retains and utilizes low‐level features. In addition, coordinate attention (CA) is integrated into the module, thereby further improving the network's ability to extract key local information. To further reduce the complexity of model training, the residual module adopts a stochastic depth strategy, providing the network with a random depth. Comparative experiments on five public datasets show that the LRSCM network can achieve results comparable to those of other state‐of‐the‐art methods.

MGML: Multigranularity Multilevel Feature Ensemble Network for Remote Sensing Scene Classification.

Remote sensing (RS) scene classification is a challenging task to predict scene categories of RS images. RS images have two main issues: large intraclass variance caused by large resolution variance and confusing information from large geographic covering area. To ease the negative influence from the above two issues. We propose a multigranularity multilevel feature ensemble network (MGML-FENet) to efficiently tackle the RS scene classification task in this article. Specifically, we propose multigranularity multilevel feature fusion branch (MGML-FFB) to extract multigranularity features in different levels of network by channel-separate feature generator (CS-FG). To avoid the interference from confusing information, we propose a multigranularity multilevel feature ensemble module (MGML-FEM), which can provide diverse predictions by full-channel feature generator (FC-FG). Compared to previous methods, our proposed networks have the ability to use structure information and abundant fine-grained features. Furthermore, through the ensemble learning method, our proposed MGML-FENets can obtain more convincing final predictions. Extensive classification experiments on multiple RS datasets (AID, NWPU-RESISC45, UC-Merced, and VGoogle) demonstrate that our proposed networks achieve better performance than previous state-of-the-art (SOTA) networks. The visualization analysis also shows the good interpretability of MGML-FENet.

Few-shot remote sensing image scene classification based on multiscale covariance metric network (MCMNet)

Few-shot learning (FSL) is a paradigm that simulates the fast learning ability of human beings, which can learn the feature differences between two groups of small-scale samples with common label space, and the label space of the training set and the test set is not repeated. By this way, it can quickly identify the categories of the unseen image in the test set. This method is widely used in image scene recognition, and it is expected to overcome difficulties of scarce annotated samples in remote sensing (RS). However, among most existing FSL methods, images were embed into Euclidean space, and the similarity between features at the last layer of deep network were measured by Euclidean distance. It is difficult to measure the inter-class similarity and intra-class difference of RS images. In this paper, we propose a multi-scale covariance network (MCMNet) for the application of remote sensing scene classification (RSSC). Taking Conv64F as the backbone, we mapped the features of the 1, 2, and 4 layers of the network to the manifold space by constructing a regional covariance matrix to form a covariance network with different scales. For each layer of features, we introduce the center in manifold space as a prototype for different categories of features. We simultaneously measure the similarity of three prototypes on the manifold space with different scales to form three loss functions and optimize the whole network by episodic training strategy. We conducted comparative experiments on three public datasets. The results show that the classification accuracy (CA) of our proposed method is from 1.35 % to 2.36% higher than that of the most excellent method, which demonstrates that the performance of MCMNet outperforms other methods.

Dual Wavelet Attention Networks for Image Classification

Global average pooling (GAP) plays an important role in traditional channel attention. However, there is the disadvantage of insufficient information to use the result of GAP as the channel scalar. At the same time, the existing spatial attention models focus on the areas of interest using average pooling or convolutional networks, but there is a loss of feature information and neglect of the structural feature. In this paper, dual wavelet attention is proposed, which can effectively alleviate the aforementioned problems and enhance the representation ability of CNNs. Firstly, the equivalence between the sum of the low-frequency subband coefficients of 2D DWT (Haar) and GAP is proved. On this basis, the statistical characteristics of low-frequency and high-frequency subbands are effectively combined to obtain the channel scalars, which can better measure the importance of each channel. In addition, 2D DWT can effectively capture the approximate and detailed structural features. Thus, wavelet spatial attention is proposed, which can effectively focus on the key spatial structural features. Different from traditional spatial attention, it can better curve the structural and spatial attention for different channels. The experiments are verified on four natural image data sets and three remote sensing scene classification data sets, which shows the effectiveness and versatility of the proposed methods. The code of this paper will be available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/yutinyang/DWAN</uri> .

P2FEViT: Plug-and-Play CNN Feature Embedded Hybrid Vision Transformer for Remote Sensing Image Classification

Remote sensing image classification (RSIC) is a classical and fundamental task in the intelligent interpretation of remote sensing imagery, which can provide unique labeling information for each acquired remote sensing image. Thanks to the potent global context information extraction ability of the multi-head self-attention (MSA) mechanism, visual transformer (ViT)-based architectures have shown excellent capability in natural scene image classification. However, in order to achieve powerful RSIC performance, it is insufficient to capture global spatial information alone. Specifically, for fine-grained target recognition tasks with high inter-class similarity, discriminative and effective local feature representations are key to correct classification. In addition, due to the lack of inductive biases, the powerful global spatial context representation capability of ViT requires lengthy training procedures and large-scale pre-training data volume. To solve the above problems, a hybrid architecture of convolution neural network (CNN) and ViT is proposed to improve the RSIC ability, called P2FEViT, which integrates plug-and-play CNN features with ViT. In this paper, the feature representation capabilities of CNN and ViT applying for RSIC are first analyzed. Second, aiming to integrate the advantages of CNN and ViT, a novel approach embedding CNN features into the ViT architecture is proposed, which can make the model synchronously capture and fuse global context and local multimodal information to further improve the classification capability of ViT. Third, based on the hybrid structure, only a simple cross-entropy loss is employed for model training. The model can also have rapid and comfortable convergence with relatively less training data than the original ViT. Finally, extensive experiments are conducted on the public and challenging remote sensing scene classification dataset of NWPU-RESISC45 (NWPU-R45) and the self-built fine-grained target classification dataset called BIT-AFGR50. The experimental results demonstrate that the proposed P2FEViT can effectively improve the feature description capability and obtain outstanding image classification performance, while significantly reducing the high dependence of ViT on large-scale pre-training data volume and accelerating the convergence speed. The code and self-built dataset will be released at our webpages.

Improving remote sensing scene classification using quality-based data augmentation

ABSTRACT This study presents an improved data augmentation technique for the classification of remote sensing (RS) scenes. The method is called Quality-based Sample Selection (QSS) data augmentation technique. It is based on generating a large number of samples using geometric transformations and then selecting the best ones based on a quality criterion. Sample images are generated online, i.e. during training of the convolutional neural network (CNN) model. For each training sample image, a few images are generated randomly using a set of geometric transformations. Then, the generated images are passed to the CNN model being trained, and the predicted probabilities are used to evaluate the quality of the images. The motivation behind this research is that we aim to augment the training set with new images that will benefit the learning of the CNN model. Therefore, an objective method to evaluate the quality of the new images is the CNN model itself. The image with the best criteria score is used to augment the training set. Several quality criteria based on the prediction probabilities of the CNN model are considered including entropy, breaking-ties and our own proposed criteria called True Label Probability (TLP). QSS is tested on five common RS scene classification datasets: UCMerced, Optimal31, RSSCN7, AID and NWPU-RS45. It has outperformed all previous methods except for one case involving NWPU-RS45 dataset. For the 10%–90% train–test split, QSS achieved 94.51, 81.72, 89.89, 94.09 and 93.98, respectively, for the mentioned datasets. While for the 20%–80% train–test split, it achieved 97.09, 88.37, 94.01, 95.71 and 94.71, respectively, for the listed datasets. Thus, this work presents a novel data augmentation method that uses data with higher quality instead of randomly selected. The paper proves experimentally that the proposed method improves the classification accuracy of RS scene classification.

Deep Semantic-Visual Alignment for zero-shot remote sensing image scene classification

Deep neural networks have achieved promising progress in remote sensing (RS) image classification, for which the training process requires abundant samples for each class. However, it is time-consuming and unrealistic to annotate labels for each RS category, given the fact that the RS target database is increasing dynamically. Zero-shot learning (ZSL) allows for identifying novel classes that are not seen during training, which provides a promising solution for the aforementioned problem. However, previous ZSL models mainly depend on manually-labeled attributes or word embeddings extracted from language models to transfer knowledge from seen classes to novel classes. Those class embeddings may not be visually detectable and the annotation process is time-consuming and labor-intensive. Besides, pioneer ZSL models use convolutional neural networks pre-trained on ImageNet, which focus on the main objects appearing in each image, neglecting the background context that also matters in RS scene classification. To address the above problems, we propose to collect visually detectable attributes automatically. We predict attributes for each class by depicting the semantic-visual similarity between attributes and images. In this way, the attribute annotation process is accomplished by machine instead of human as in other methods. Moreover, we propose a Deep Semantic-Visual Alignment (DSVA) that take advantage of the self-attention mechanism in the transformer to associate local image regions together, integrating the background context information for prediction. The DSVA model further utilizes the attribute attention maps to focus on the informative image regions that are essential for knowledge transfer in ZSL, and maps the visual images into attribute space to perform ZSL classification. With extensive experiments, we show that our model outperforms other state-of-the-art models by a large margin on a challenging large-scale RS scene classification benchmark. Moreover, we qualitatively verify that the attributes annotated by our network are both class discriminative and semantic related, which benefits the zero-shot knowledge transfer.

Credible Remote Sensing Scene Classification Using Evidential Fusion on Aerial-Ground Dual-View Images

Due to their ability to offer more comprehensive information than data from a single view, multi-view (e.g., multi-source, multi-modal, multi-perspective) data are being used more frequently in remote sensing tasks. However, as the number of views grows, the issue of data quality is becoming more apparent, limiting the potential benefits of multi-view data. Although recent deep neural network (DNN)-based models can learn the weight of data adaptively, a lack of research on explicitly quantifying the data quality of each view when fusing them renders these models inexplicable, performing unsatisfactorily and inflexibly in downstream remote sensing tasks. To fill this gap, in this paper, evidential deep learning is introduced to the task of aerial-ground dual-view remote sensing scene classification to model the credibility of each view. Specifically, the theory of evidence is used to calculate an uncertainty value which describes the decision-making risk of each view. Based on this uncertainty, a novel decision-level fusion strategy is proposed to ensure that the view with lower risk obtains more weight, making the classification more credible. On two well-known, publicly available datasets of aerial-ground dual-view remote sensing images, the proposed approach achieves state-of-the-art results, demonstrating its effectiveness.

Subspace prototype learning for few-Shot remote sensing scene classification

Recently, few-shot remote sensing scene classification (FSRSSC)has attracted more and more researchers’ attention. The FSRSSC aims to solve the problem of how to distinguish novel categories when there are limited labeled samples quickly. The FSRSSC framework mainly includes two stages: 1) pre-training (meta-training) stage, which uses training data to train the feature extractor. 2) meta-testing stage, the trained feature extractor is used to extract the features of testing data of different categories from the training data. A limited number of labeled samples are used to train the classifier and complete the classification task. In this paper, we proposed subspace prototype learning for few-shot remote sensing scene classification method (SPL). To improve the generalization performance of the feature extractor in the pre-training stage and the robustness of the classifier in the meta-testing stage, we improve the existing methods from two aspects. On the one hand, we introduce the pre-trained model on the natural images. Then we use remote sensing data to fine-tune the pre-trained model to solve the problem of negative transfer caused by the differences between natural and remote sensing images. On the other hand, in the meta-test phase, we use the subspace learning method to learn a subspace prototype for each type of sample and complete the classification task by measuring the distance between the samples and the prototype, which achieves performance classification performance.

A coupled multi-task feature boosting method for remote sensing scene classification

A coupled multi-task feature boosting method for remote sensing scene classification

Adversarial Remote Sensing Scene Classification Based on Lie Group Feature Learning

Convolutional Neural Networks have been widely used in remote sensing scene classification. Since this kind of model needs a large number of training samples containing data category information, a Generative Adversarial Network (GAN) is usually used to address the problem of lack of samples. However, GAN mainly generates scene data samples that do not contain category information. To address this problem, a novel supervised adversarial Lie Group feature learning network is proposed. In the case of limited data samples, the model can effectively generate data samples with category information. There are two main differences between our method and the traditional GAN. First, our model takes category information and data samples as the input of the model and optimizes the constraint of category information in the loss function, so that data samples containing category information can be generated. Secondly, the object scale sample generation strategy is introduced, which can generate data samples of different scales and ensure that the generated data samples contain richer feature information. After large-scale experiments on two publicly available and challenging datasets, it is found that our method can achieve better scene classification accuracy even with limited data samples.

HCFPN: Hierarchical Contextual Feature-Preserved Network for Remote Sensing Scene Classification

Convolutional neural networks (CNNs) have made significant advances in remote sensing scene classification (RSSC) in recent years. Nevertheless, the limitations of the receptive field cause CNNs to suffer from a disadvantage in capturing contextual information. To address this issue, vision transformer (ViT), a novel model that has piqued the interest of academics, is used to extract latent contextual information in remote sensing scene classification. However, when confronted with the challenges of large-scale variations and high interclass similarity in scene classification images, the original ViT has the drawback of ignoring important local features, thereby causing the model’s performance to degrade. Consequently, we propose the hierarchical contextual feature-preserved network (HCFPN) by combining the advantages of CNNs and ViT. First, a hierarchical feature extraction module based on ResNet-34 is utilized to acquire the multilevel convolutional features and high-level semantic features. Second, a contextual feature-preserved module takes advantage of the first two multilevel features to capture abundant long-term contextual features. Then, the captured long-term contextual features are utilized for multiheaded cross-level attention computing to aggregate and explore the correlation of multilevel features. Finally, the multiheaded cross-level attention score and high-level semantic features are classified. Then, a category score average module is proposed to fuse the classification results, whereas a label smoothing approach is utilized prior to calculating the loss to produce discriminative scene representation. In addition, we conduct extensive experiments on two publicly available RSSC datasets. Our proposed HCPFN outperforms most state-of-the-art approaches.

Dictionary Learning for Few-Shot Remote Sensing Scene Classification

With deep learning-based methods growing (even with scarce data in some fields), few-shot remote sensing scene classification (FSRSSC) has received a lot of attention. One mainstream approach uses base data to train a feature extractor (FE) in the pre-training phase and employs novel data to design the classifier and complete the classification task in the meta-test phase. Due to the scarcity of remote sensing data, obtaining a suitable feature extractor for remote sensing data and designing a robust classifier have become two major challenges. In this paper, we propose a novel dictionary learning (DL) algorithm for few-shot remote sensing scene classification to address these two difficulties. First, we use natural image datasets with sufficient data to obtain a pre-trained feature extractor. We fine-tune the parameters with the remote sensing dataset to make the feature extractor suitable for remote sensing data. Second, we design the kernel space classifier to map the features to a high-dimensional space and embed the label information into the dictionary learning to improve the discrimination of features for classification. Extensive experiments on four popular remote sensing scene classification datasets demonstrate the effectiveness of our proposed dictionary learning method.

BiShuffleNeXt: A lightweight bi-path network for remote sensing scene classification

To address the problem of excessive computation in the remote sensing scene classification (RSSC) algorithms, this paper proposes an efficient RSSC model, which is named BiShuffleNeXt. Firstly, we use the sandglass bottleneck to design the ShuffleNeXt as a context path for deep semantic information extraction. Secondly, the spatial path consists of three sandglass bottlenecks, each with a stride of 2, to retain more spatial information and obtain a high-resolution feature map. Finally, the feature combination module is designed to perform the fusion of bi-path features. Experimental results show that the proposed model achieves a classification accuracy of 95.93% on the NWPU-45 dataset, which is a 3.6% improvement over using the context path alone. The proposed method is tested on three datasets and outperforms ShuffleNet v2, MobileNetv2, Mobile-Former and ResNet-50 respectively. The floating point operations (FLOPs) of BiShuffleNeXt are only 298.25M, which greatly reduces the calculation amount while ensuring classification accuracy.

Adaptive Discriminative Regions Learning Network for Remote Sensing Scene Classification.

As an auxiliary means of remote sensing (RS) intelligent interpretation, remote sensing scene classification (RSSC) attracts considerable attention and its performance has been improved significantly by the popular deep convolutional neural networks (DCNNs). However, there are still several challenges that hinder the practical applications of RSSC, such as complex composition of land cover, scale-variation of objects, and redundant and noisy areas for scene classification. In order to mitigate the impact of these issues, we propose an adaptive discriminative regions learning network for RSSC, referred as ADRL-Net briefly, which locates discriminative regions effectively for boosting the performance of RSSC by utilizing a novel self-supervision mechanism. Our proposed ADRL-Net consists of three main modules, including a discriminative region generator, a region discriminator, and a region scorer. Specifically, the discriminative region generator first generates some candidate regions which could be informative for RSSC. Then, the region discriminator evaluates the regions generated by region generator and provides feedback for the generator to update the informative regions. Finally, the region scorer makes prediction scores for the whole image by using the discriminative regions. In such a manner, the three modules of ADRL-Net can cooperate with each other and focus on the most informative regions of an image and reduce the interference of redundant regions for final classification, which is robust to the complex scene composition, object scales, and irrelevant information. In order to validate the efficacy of the proposed network, we conduct experiments on four widely used benchmark datasets, and the experimental results demonstrate that ADRL-Net consistently outperforms other state-of-the-art RSSC methods.