Aerial Image Dataset Research Articles

Autonomous Drone Navigation Using Computer Vision

Abstract: Autonomous drone navigation with computer vision is an innovative technology that allows drones to move through intricate surroundings without needing human control. This system uses live visual information to identify objects, steer clear of obstacles, and determine routes, improving operational safety and precision. The project's main objective is to create a visionbased navigation system that combines object detection and obstacle avoidance algorithms through deep learning methods like YOLO (You Only Look Once), alongside real-time sensor fusion. The drone uses computer vision algorithms to process aerial images and automatically changes its flight path to prevent crashes. A personalized dataset of aerial images is generated and utilized for improving object detection skills in various environments. In order to guarantee practicality in real- world situations, the system undergoes validation through simulations and field tests on different terrains, focusing on its resilience in changing environments. Improvements in navigation accuracy and obstacle detection are accomplished by implementing adaptive pathplanning and integrating multiple sensors, guaranteeing the drone's efficient operation in real-life situations. The goal of this method is to enhance the drone's ability to make decisions, minimize human mistakes, and expand its range of potential uses in areas like surveillance, agriculture, and disaster response

Variational Autoencoder with Gaussian Random Field prior: Application to unsupervised animal detection in aerial images

In real world datasets of aerial images, the objects of interest are often missing, hard to annotate and of varying aspects. The framework of unsupervised Anomaly Detection (AD) is highly relevant in this context, and Variational Autoencoders (VAEs), a family of popular probabilistic models, are often used. We develop on the literature of VAEs for AD in order to take advantage of the particular textures that appear in natural aerial images. More precisely we propose a new VAE model with a Gaussian Random Field (GRF) prior (VAE-GRF), which generalizes the classical VAE model, and we provide the necessary procedures and hypotheses required for the model to be tractable. We show that, under some assumptions, the VAE-GRF largely outperforms the traditional VAE and some other probabilistic models developed for AD. Our results suggest that the VAE-GRF could be used as a relevant VAE baseline in place of the traditional VAE with very limited additional computational cost. We provide competitive results on the MVTec reference dataset for visual inspection, and two other datasets dedicated to the task of unsupervised animal detection in aerial images.

Teacher-Student Prototype Enhancement Network for a Few-Shot Remote Sensing Scene Classification

Few-shot remote sensing scene classification identifies new classes from limited labeled samples where the great challenges are intraclass diversity, interclass similarity, and limited supervision. To alleviate these problems, a teacher-student prototype enhancement network is proposed for a few-shot remote sensing scene classification. Instead of introducing an attentional mechanism in mainstream studies, a prototype enhancement module is recommended to adaptively select high-confidence query samples, which can enhance the support prototype representations to emphasize intraclass and interclass relationships. The construction of a few-shot teacher model generates more discriminative predictive representations with inputs from many labeled samples, thus providing a strong supervisory signal to the student model and encouraging the network to achieve accurate classification with a limited number of labeled samples. Extensive experiments of four public datasets, including NWPU-remote sens ing image scene classification (NWPU-RESISC45), aerial image dataset (AID), UC Merced, and WHU-RS19, demonstrate that this method achieves superior competitive performance than the state-of-the-art methods on five-way, one-shot, and five-shot classifications.

Crop Growth Analysis Using Automatic Annotations and Transfer Learning in Multi-Date Aerial Images and Ortho-Mosaics

Growth monitoring of crops is a crucial aspect of precision agriculture, essential for optimal yield prediction and resource allocation. Traditional crop growth monitoring methods are labor-intensive and prone to errors. This study introduces an automated segmentation pipeline utilizing multi-date aerial images and ortho-mosaics to monitor the growth of cauliflower crops (Brassica Oleracea var. Botrytis) using an object-based image analysis approach. The methodology employs YOLOv8, a Grounding Detection Transformer with Improved Denoising Anchor Boxes (DINO), and the Segment Anything Model (SAM) for automatic annotation and segmentation. The YOLOv8 model was trained using aerial image datasets, which then facilitated the training of the Grounded Segment Anything Model framework. This approach generated automatic annotations and segmentation masks, classifying crop rows for temporal monitoring and growth estimation. The study’s findings utilized a multi-modal monitoring approach to highlight the efficiency of this automated system in providing accurate crop growth analysis, promoting informed decision-making in crop management and sustainable agricultural practices. The results indicate consistent and comparable growth patterns between aerial images and ortho-mosaics, with significant periods of rapid expansion and minor fluctuations over time. The results also indicated a correlation between the time and method of observation which paves a future possibility of integration of such techniques aimed at increasing the accuracy in crop growth monitoring based on automatically derived temporal crop row segmentation masks.

Measuring ornamental tree canopy attributes for precision spraying using drone technology and self-supervised segmentation

Tree canopy attributes or characteristics, such as canopy volume and density, are important parameters for calculating the precise quantity of agrochemicals required in each tree. Assessing the agrochemical needs in a nursery can help growers effectively estimate expenses, mitigate the risks of overuse or underuse, enhance resource utilization, and promote sustainable agricultural practices. This study aimed to develop an unmanned aerial vehicle (UAV)-based system to measure tree canopy attributes like tree height, tree count, canopy area and canopy volume using an image processing and self-supervised zero-shot segmentation approach. A high-resolution red-green-blue (RGB) sensor mounted on a drone captured three aerial image datasets, D1 and D2 from the first experimental plot and D3 from the second experimental plot. The acquired aerial images were stitched together and processed to generate a digital surface model (DSM) and a digital terrain model (DTM). A self-supervised zero-shot segmentation model, Segment Anything Model (SAM), was applied for tree segmentation and counting. The tree canopy area was calculated by segmenting individual trees and applying a conversion factor to convert pixelwise areas into square meters. Height calculation was done using elevation data of each pixel from DSM-DTM imagery within SAM-derived bounding box coordinates, and the highest pixel was considered as the height of the tree. The drone-calculated heights of 24 randomly selected trees were compared with manually measured heights in all datasets. The results showed an average absolute error of 2.43% in D1, 7.09% in D2 and 12.58% in D3. The Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) were noted to be 0.09 m and 0.23 m in D1, 0.13 m and 0.25 m in D2, and 0.16 m and 0.21 m in D3, respectively. In D3, manual area and volume calculations were performed for validation. The RMSE and MAE for area calculation were calculated as 0.18 m2 and 0.15 m2, respectively, and the RMSE and MAE for volume measurements were calculated as 0.33 m3 and 0.26 m3 respectively, which indicated the level of agreement between manual and drone measurements. Using canopy volume information, calculated by summing pixel heights within each tree’s bounding box and multiplying by the ground sample distance (GSD) of 0.025 m per pixel, the average quantity of agrochemicals needed for a Maple tree was estimated at 0.20 L. These results underscore the high potential of this method for accurate canopy characteristic calculations and precision spray applications in the ornamental industry.

SDMNet: Spatially dilated multi-scale network for object detection for drone aerial imagery

Multi-scale object detection is a preeminent challenge in computer vision and image processing. Several deep learning models that are designed to detect various objects miss out on the detection capabilities for small objects, reducing their detection accuracies. Intending to focus on different scales, from extremely small to large-sized objects, this work proposes a Spatially Dilated Multi-Scale Network (SDMNet) architecture for UAV-based ground object detection. It proposes a Multi-scale Enhanced Effective Channel Attention mechanism to preserve the object details in the images. Additionally, the proposed model incorporates dilated convolution, sub-pixel convolution, and additional prediction heads to enhance object detection performance specifically for aerial imaging. It has been evaluated on two popular aerial image datasets, VisDrone 2019 and UAVDT, containing publicly available annotated images of ground objects captured from UAV. Different performance metrics, such as precision, recall, mAP, and detection rate, benchmark the proposed architecture with the existing object detection approaches. The experimental results demonstrate the effectiveness of the proposed model for multi-scale object detection with an average precision score of 54.2% and 98.4% for VisDrone and UAVDT datasets, respectively.

Restoration of Semantic-Based Super-Resolution Aerial Images

Currently, technologies for remote sensing image processing are actively developing, including both satellite images and aerial images obtained from video cameras of unmanned aerial vehicles. Often such images have artifacts such as low resolution, blurred image fragments, noise, etc. One way to overcome such limitations is to use modern technologies to restore super-resolution images based on deep learning methods. The specificity of aerial images is the presentation of texture and structural elements in a higher resolution than in satellite images, which objectively contributes to better results of restoration. The article provides a classification of super-resolution methods based on the main architectures of deep neural networks, namely convolutional neural networks, visual transformers and generative adversarial networks. The article proposes a method for reconstructing super-resolution aerial images SemESRGAN taking into account semantic features by using an additional deep network for semantic segmentation during the training stage. The total loss function, including adversarial losses, pixel-level losses, and perception losses (feature similarity), is minimized. Six annotated aerial and satellite image datasets CLCD, DOTA, LEVIR-CD, UAVid, AAD, and AID were used for the experiments. The results of image restoration using the proposed SemESRGAN method were compared with the basic architectures of convolutional neural networks, visual transformers and generative adversarial networks. Comparative results of image restoration were obtained using objective metrics PSNR and SSIM, which made it possible to evaluate the quality of restoration using various deep network models.

Efficient geospatial mapping of buildings, woodlands, water and roads from aerial imagery using deep learning.

As more aerial imagery becomes readily available, massive volumes of data are being gathered constantly. Several groups can benefit from the data provided by this geographical imagery. However, it is time-consuming to manually analyze each image to gain information on land cover. This research suggests using deep learning methods for precise and rapid pixel-by-pixel classification of aerial imagery for land cover analysis, which would be a significant step forward in resolving this issue. The suggested method has several steps, such as the augmentation and transformation of data, the selection of deep learning models, and the final prediction. The study uses the three most popular deep learning models (Vanilla-UNet, ResNet50 UNet, and DeepLabV3 ResNet50) for the experiments. According to the experimental results, the ResNet50 UNet model achieved an accuracy of 94.37%, the DeepLabV3 ResNet50 model achieved an accuracy of 94.77%, and the Vanilla-UNet model achieved an accuracy of 91.31%. The accuracy, precision, recall, and F1-score of DeepLabV3 and ResNet50 are higher than those of the other two models. The proposed approach is also compared to the existing UNet approach, and the proposed approaches have produced greater probability prediction scores than the conventional UNet model for all classes. Our approach outperforms model DeepLabV3 ResNet50 on aerial image datasets based on the performance.

Landslide extraction from aerial imagery considering context association characteristics

Accurate extraction of landslide information is crucial for timely disaster emergency response, yet this process faces significant challenges due to the interference of bare land and vegetation cover as well as the variability in landslide scales. Furthermore, the scarcity of comprehensive landslide datasets has hindered the in-depth exploration of deep learning techniques in this domain. Addressing these issues, this study introduces a novel framework for landslide extraction that leverages the characteristics of context association. In the encoding phase, a two-branch multiscale context feature extraction module (TMCFM) is established, which captures the contextual relationships across different scales through an attention mechanism while concurrently extracting context information within the same scale. To refine the interplay between features at different levels, self-attention is utilized, allowing for the effective fusion of context information at both ends of the same scale through feature fusion exploiting static and dynamic context feature modules (FSDC). Building upon this, this study develops a deeply supervised classifier (DSC) that enhances the network’s discriminative capabilities in the prediction phase via six auxiliary branches. Additionally, this study contributes a new aerial image dataset for landslide extraction created through meticulous visual interpretation. The proposed method is thoroughly compared with 17 contemporary deep learning methods, demonstrating an increase in the intersection-over-union (IoU) metric of 0.92%-16.94% over these methods. The robustness and superiority of the proposed approach are further validated through various discussions and analyses.

ASG-YOLOv5: Improved YOLOv5 unmanned aerial vehicle remote sensing aerial images scenario for small object detection based on attention and spatial gating.

With the accelerated development of the technological power of society, aerial images of drones gradually penetrated various industries. Due to the variable speed of drones, the captured images are shadowed, blurred, and obscured. Second, drones fly at varying altitudes, leading to changing target scales and making it difficult to detect and identify small targets. In order to solve the above problems, an improved ASG-YOLOv5 model is proposed in this paper. Firstly, this research proposes a dynamic contextual attention module, which uses feature scores to dynamically assign feature weights and output feature information through channel dimensions to improve the model's attention to small target feature information and increase the network's ability to extract contextual information; secondly, this research designs a spatial gating filtering multi-directional weighted fusion module, which uses spatial filtering and weighted bidirectional fusion in the multi-scale fusion stage to improve the characterization of weak targets, reduce the interference of redundant information, and better adapt to the detection of weak targets in images under unmanned aerial vehicle remote sensing aerial photography; meanwhile, using Normalized Wasserstein Distance and CIoU regression loss function, the similarity metric value of the regression frame is obtained by modeling the Gaussian distribution of the regression frame, which increases the smoothing of the positional difference of the small targets and solves the problem that the positional deviation of the small targets is very sensitive, so that the model's detection accuracy of the small targets is effectively improved. This paper trains and tests the model on the VisDrone2021 and AI-TOD datasets. This study used the NWPU-RESISC dataset for visual detection validation. The experimental results show that ASG-YOLOv5 has a better detection effect in unmanned aerial vehicle remote sensing aerial images, and the frames per second (FPS) reaches 86, which meets the requirement of real-time small target detection, and it can be better adapted to the detection of the weak and small targets in the aerial image dataset, and ASG-YOLOv5 outperforms many existing target detection methods, and its detection accuracy reaches 21.1% mAP value. The mAP values are improved by 2.9% and 1.4%, respectively, compared with the YOLOv5 model. The project is available at https://github.com/woaini-shw/asg-yolov5.git.

Algal Bed Region Segmentation Based on a ViT Adapter Using Aerial Images for Estimating CO2 Absorption Capacity

In this study, we propose a novel method for algal bed region segmentation using aerial images. Accurately determining the carbon dioxide absorption capacity of coastal algae requires measurements of algal bed regions. However, conventional manual measurement methods are resource-intensive and time-consuming, which hinders the advancement of the field. To solve these problems, we propose a novel method for automatic algal bed region segmentation using aerial images. In our method, we use an advanced semantic segmentation model, a ViT adapter, and adapt it to aerial images for algal bed region segmentation. Our method demonstrates high accuracy in identifying algal bed regions in an aerial image dataset collected from Hokkaido, Japan. The experimental results for five different ecological regions show that the mean intersection over union (mIoU) and mean F-score of our method in the validation set reach 0.787 and 0.870, the IoU and F-score for the background region are 0.957 and 0.978, and the IoU and F-score for the algal bed region are 0.616 and 0.762, respectively. In particular, the mean recognition area compared with the ground truth area annotated manually is 0.861. Our study contributes to the advancement of blue carbon assessment by introducing a novel semantic segmentation-based method for identifying algal bed regions using aerial images.

Calibrating an airborne linear-array multi-camera system on the master focal plane with existing bundled images

ABSTRACT Integrating multiple hyperspectral linear-array cameras into a Linear-array Multi-camera System (LMS) enables an expanded Field of View (FOV) while preserving high spatial resolution. However, the constrained capabilities of tie points are weakened by the limited overlap and short baseline between sub-cameras, hindering the attainment of reliable solutions for the theoretically ideal relative calibration model. Additionally, the absence of stereoscopic imaging in the LMS introduces the challenge of acquiring dense Ground Control Points (GCPs) throughout the FOV during calibration. In this paper, for calibrating an advanced airborne hyperspectral LMS named Arial Hyperspectral System-1000 (AHS-1000), we propose an equivalent geometric model that transforms the three-camera configuration of the LMS into a single-camera representation on the master camera’s focal plane, which was successfully applied to the data processing system of AHS-1000. We extracted dense GCPs by matching each Level-1 sub-image of the LMS with the bundled images of an existing three-line-scanner aerial image dataset, whose orientation parameters have been accurately solved by bundle adjustment with sparse GCPs; the matched points of each sub-image were consistently projected onto the master focal plane as image observations for our equivalent model. Finally, we established a collinearity equation system and performed a combined strip adjustment to calibrate the distortion terms for each pixel of the equivalent camera. The results demonstrate the effectiveness and robustness of our equivalent modeling method, which outperforms the classic modeling method by 2.83 pixels in interior accuracy and 0.86 pixels in relative accuracy, as well as outperforming another similar equivalent modeling method by 1.22 pixels in interior accuracy and 0.18 pixels in relative accuracy. Additionally, our comparison utilizing bundled images, public DOM (Digital Orthophoto Maps) /DEM (Digital Elevation Models), and high-precision DOM/DEM from aerial three-line-scanner demonstrates that bundled images can achieve higher calibration accuracy.

Multi-Scale Feature Fusion Attention Network for Building Extraction in Remote Sensing Images

The efficient semantic segmentation of buildings in high spatial resolution remote sensing images is a technical prerequisite for land resource management, high-precision mapping, construction planning and other applications. Current building extraction methods based on deep learning can obtain high-level abstract features of images. However, the extraction of some occluded buildings is inaccurate, and as the network deepens, small-volume buildings are lost and edges are blurred. Therefore, we introduce a multi-resolution attention combination network, which employs a multiscale channel and spatial attention module (MCAM) to adaptively capture key features and eliminate irrelevant information, which improves the accuracy of building extraction. In addition, we present a layered residual connectivity module (LRCM) to enhance the expression of information at different scales through multi-level feature fusion, significantly improving the understanding of context and the capturing of fine edge details. Extensive experiments were conducted on the WHU aerial image dataset and the Massachusetts building dataset. Compared with state-of-the-art semantic segmentation methods, this network achieves better building extraction results in remote sensing images, proving the effectiveness of the method.

A modified generative adversarial networks with Yolov5 for automated forest health diagnosis from aerial imagery and Tabu search algorithm.

Our environment has been significantly impacted by climate change. According to previous research, insect catastrophes induced by global climate change killed many trees, inevitably contributing to forest fires. The condition of the forest is an essential indicator of forest fires. Analysis of aerial images of a forest can detect deceased and living trees at an early stage. Automated forest health diagnostics are crucial for monitoring and preserving forest ecosystem health. Combining Modified Generative Adversarial Networks (MGANs) and YOLOv5 (You Only Look Once version 5) is presented in this paper as a novel method for assessing forest health using aerial images. We also employ the Tabu Search Algorithm (TSA) to enhance the process of identifying and categorizing unhealthy forest areas. The proposed model provides synthetic data to supplement the limited labeled dataset, thereby resolving the frequent issue of data scarcity in forest health diagnosis tasks. This improvement enhances the model's ability to generalize to previously unobserved data, thereby increasing the overall precision and robustness of the forest health evaluation. In addition, YOLOv5 integration enables real-time object identification, enabling the model to recognize and pinpoint numerous tree species and potential health issues with exceptional speed and accuracy. The efficient architecture of YOLOv5 enables it to be deployed on devices with limited resources, enabling forest-monitoring applications on-site. We use the TSA to enhance the identification of unhealthy forest areas. The TSA method effectively investigates the search space, ensuring the model converges to a near-optimal solution, improving disease detection precision and decreasing false positives. We evaluated our MGAN-YOLOv5 method using a large dataset of aerial images of diverse forest habitats. The experimental results demonstrated impressive performance in diagnosing forest health automatically, achieving a detection precision of 98.66%, recall of 99.99%, F1 score of 97.77%, accuracy of 99.99%, response time of 3.543ms and computational time of 5.987ms. Significantly, our method outperforms all the compared target detection methods showcasing a minimum improvement of 2% in mAP.

Remote sensing scene classification using multi-domain sematic high-order network

Recently, convolutional neural networks (CNNs), which obtain powerful deep features in an end-to-end manner, have achieved powerful performance in remote sensing scene classification. However, the average or maximum pooling operations defined in the spatial domain and coarser-resolution features with high levels cannot extract reliable features and clear boundaries for small-scale targets in remote sensing scene imagery. This paper attempts to address these problems and proposes a multi-domain sematic high-order network for scene classification, named MSHNet. First, wavelet-spatial and detachable pooling blocks defined in the wavelet and spatial domains are inserted at the end of the convolutional block to learn the features in a more structural fusion manner. Second, multi-scale and multi-resolution semantic embedding modules are proposed to take full advantage of the complementary information and effectively maintain the spatial structures of learned deep features. Third, we employ a factorized bilinear coding approach to obtain compact and discriminative second-order features. MSHNet is thoroughly evaluated on two publicly available benchmarks, i.e., AID (Aerial Image Dataset) and NWPU-RESISC45 (Northwestern Polytechnical University-Remote Sensing Image Scene Classification 45). The extensive results illustrate that our MSHNet is competitive with other related multi-scale deep neural networks.

Multi-Scale Attention Network for Building Extraction from High-Resolution Remote Sensing Images

The precise building extraction from high-resolution remote sensing images holds significant application for urban planning, resource management, and environmental conservation. In recent years, deep neural networks (DNNs) have garnered substantial attention for their adeptness in learning and extracting features, becoming integral to building extraction methodologies and yielding noteworthy performance outcomes. Nonetheless, prevailing DNN-based models for building extraction often overlook spatial information during the feature extraction phase. Additionally, many existing models employ a simplistic and direct approach in the feature fusion stage, potentially leading to spurious target detection and the amplification of internal noise. To address these concerns, we present a multi-scale attention network (MSANet) tailored for building extraction from high-resolution remote sensing images. In our approach, we initially extracted multi-scale building feature information, leveraging the multi-scale channel attention mechanism and multi-scale spatial attention mechanism. Subsequently, we employed adaptive hierarchical weighting processes on the extracted building features. Concurrently, we introduced a gating mechanism to facilitate the effective fusion of multi-scale features. The efficacy of the proposed MSANet was evaluated using the WHU aerial image dataset and the WHU satellite image dataset. The experimental results demonstrate compelling performance metrics, with the F1 scores registering at 93.76% and 77.64% on the WHU aerial imagery dataset and WHU satellite dataset II, respectively. Furthermore, the intersection over union (IoU) values stood at 88.25% and 63.46%, surpassing benchmarks set by DeepLabV3 and GSMC.

Land scene classification from remote sensing images using improved artificial bee colony optimization algorithm

The images obtained from remote sensing consist of background complexities and similarities among the objects that act as challenge during the classification of land scenes. Land scenes are utilized in various fields such as agriculture, urbanization, and disaster management, to detect the condition of land surfaces and help to identify the suitability of the land surfaces for planting crops, and building construction. The existing methods help in the classification of land scenes through the images obtained from remote sensing technology, but the background complexities and presence of similar objects act as a barricade against providing better results. To overcome these issues, an improved artificial bee colony optimization algorithm with convolutional neural network (IABC-CNN) model is proposed to achieve better results in classifying the land scenes. The images are collected from aerial image dataset (AID), Northwestern Polytechnical University-Remote Sensing Image Scene 45 (NWPU-RESIS45), and University of California Merced (UCM) datasets. IABC effectively selects the best features from the extracted features using visual geometry group-16 (VGG-16). The selected features from the IABC are provided for the classification process using multiclass-support vector machine (MSVM). Results obtained from the proposed IABC-CNN achieves a better classification accuracy of 96.40% with an error rate 3.6%.

Geolocalization from Aerial Sensing Images Using Road Network Alignment

Estimating the geographic positions in GPS-denied environments is of great significance to the safe flight of unmanned aerial vehicles (UAVs). In this paper, we propose a novel geographic position estimation method for UAVs after road network alignment. We discuss the generally overlooked issue, namely, how to estimate the geographic position of the UAV after successful road network alignment, and propose a precise robust solution. In our method, the optimal initial solution of the geographic position of the UAV is first estimated from the road network alignment result, which is typically presented as a homography transformation between the observed road map and the reference one. The geographic position estimation is then modeled as an optimization problem to align the observed road with the reference one to improve the estimation accuracy further. Experiments on synthetic and real flight aerial image datasets show that the proposed algorithm can estimate more accurate geographic position of the UAV in real time and is robust to the errors from homography transformation estimation compared to the currently commonly-used method.

MSE-Net: A novel master–slave encoding network for remote sensing scene classification

Remote sensing scene (RSS) image classification plays a vital role in various fields such as urban planning and environmental protection. However, due to higher inter-class similarity and intra-class variability, achieving accurate classification for RSS images poses a considerable challenge for current convolutional neural networks (CNNs)-based and visual transformer (ViT)-based methods. To address these issues, this paper proposes a novel dual-encoding method named master–slave encoding network (MSE-Net) from two perspectives of feature extraction and fusion. The master encoder, based on ViT, extracts higher-level semantic features, while the slave encoder, based on CNN, captures relative lower-level spatial structure information. Secondly, to integrate feature information from the two encoders effectively, this paper further develop two fusion strategies. The first strategy involves the auxiliary enhancement units (AEUs), which eliminates semantic divergence between the two encoders, enhances spatial context awareness of the slave encoder and promotes effective feature learning. The interactive perception unit (IPU), as the second strategy, facilitates interaction and integration of the two encoders’ representations to extract more discriminative feature information. In addition, we conducted comparative experiments on four widely-used RSS datasets, including RSSCN7, SIRI-WHU, the aerial image dataset (AID) and NWPU-RESISC45 (NWPU45), to verify the effectiveness of MSE-Net. The experimental results demonstrate that MSE-Net achieved state-of-the-art (SOTA) performance across all the datasets.

A hybrid CNN–vision transformer structure for remote sensing scene classification

ABSTRACT Vision Transformers (ViTs) have become one of the main architectures in deep learning with the self-attention mechanism, and are becoming an alternative to Convolutional Neural Networks (CNNs) for remote sensing scene classification tasks. However, the earlier self-attention layer of ViTs focuses on local features rather than global features, and the deeper self-attention layer focuses on global features but ignores the different impact of different frequency information. This will greatly increase the training and computational cost due to the quadratic complexity of the self-attention mechanism on the long sequence representation. In this paper, we propose a hybrid CNN – vision transformer structure (HCVNet), which uses convolutional layers to replace the earlier self-attention layers, and a novel Frequency Multi-head Self Attention (F-MSA) mechanism to replace the deeper self-attention layers. Specifically, F-MSA is a dual-stream structure that reduces the computational cost and improves the classification performance by encoding the high/low frequency information separately. In addition, a Semantic-aware Localization (SaL) module is introduced, which can guide the selection of crop by learning prior knowledge, avoiding the issue of pure background sampling. Our method performed an accuracy value of 97.20 ± 0.02% on the Aerial Image Dataset and 93.89 ± 0.03% on the NWPU-RESISC45 Dataset, with low complexity costs.