Popular Deep Learning Frameworks Research Articles

Exploring the effects of RNNs and deep learning frameworks on real‐time, lightweight, adaptive time series anomaly detection

SummaryReal‐time, lightweight, adaptive time series anomaly detection is increasingly critical in cybersecurity, industrial control, finance, healthcare, and many other domains due to its capability to promptly process time series and detect anomalies without requiring extensive computation resources. While numerous anomaly detection approaches have emerged recently, they generally employ a single type of recurrent neural network (RNN) and are implemented using a single type of deep learning framework. The impacts of using various RNN types across different deep learning frameworks on the performance of these approaches remain unclear due to a lack of comprehensive evaluations. In this article, we aim to investigate the impact of different RNN variants and deep learning frameworks on real‐time, lightweight, and adaptive time series anomaly detection. We reviewed several state‐of‐the‐art anomaly detection approaches and implemented a representative approach using several RNN variants supported by three popular deep learning frameworks. A thorough evaluation was conducted to analyze the detection accuracy, time efficiency, and resource consumption of each implementation using four real‐world, open‐source time series datasets. The results show that RNN variants and deep learning frameworks have a significant impact. Therefore, it is crucial to carefully select appropriate RNN variants and deep learning frameworks for the implementation.

Use of Deep-Learning-Accelerated Gradient Approximation for Reservoir Geological Parameter Estimation

The estimation of space-varying geological parameters is often not computationally affordable for high-dimensional subsurface reservoir modeling systems. The adjoint method is generally regarded as an efficient approach for obtaining analytical gradient and, thus, proceeding with the gradient-based iteration algorithm; however, the infeasible memory requirement and computational demands strictly prohibit its generic implementation, especially for high-dimensional problems. The autoregressive neural network (aNN) model, as a nonlinear surrogate approximation, has gradually received increasing popularity due to significant reduction of computational cost, but one prominent limitation is that the generic application of aNN to large-scale reservoir models inevitably poses challenges in the training procedure, which remains unresolved. To address this issue, model-order reduction could be a promising strategy, which enables us to train the neural network in a very efficient manner. A very popular projection-based linear reduction method, i.e., propel orthogonal decomposition (POD), is adopted to achieve dimensionality reduction. This paper presents an architecture of a projection-based autoregressive neural network that efficiently derives an easy-to-use adjoint model by the use of an auto-differentiation module inside the popular deep learning frameworks. This hybrid neural network proxy, referred to as POD-aNN, is capable of speeding up derivation of reduced-order adjoint models. The performance of POD-aNN is validated through a synthetic 2D subsurface transport model. The use of POD-aNN significantly reduces the computation cost while the accuracy remains. In addition, our proposed POD-aNN can easily obtain multiple posterior realizations for uncertainty evaluation. The developed POD-aNN emulator is a data-driven approach for reduced-order modeling of nonlinear dynamic systems and, thus, should be a very efficient modeling tool to address many engineering applications related to intensive simulation-based optimization.

Porous-DeepONet: Learning the Solution Operators of Parametric Reactive Transport Equations in Porous Media

Reactive transport equations in porous media are critical in various scientific and engineering disciplines, but solving these equations can be computationally expensive when exploring different scenarios, such as varying porous structures and initial or boundary conditions. The deep operator network (DeepONet) has emerged as a popular deep learning framework for solving parametric partial differential equations. However, applying the DeepONet to porous media presents significant challenges due to its limited capability to extract representative features from intricate structures. To address this issue, we propose the Porous-DeepONet, a simple yet highly effective extension of the DeepONet framework that leverages convolutional neural networks (CNNs) to learn the solution operators of parametric reactive transport equations in porous media. By incorporating CNNs, we can effectively capture the intricate features of porous media, enabling accurate and efficient learning of the solution operators. We demonstrate the effectiveness of the Porous-DeepONet in accurately and rapidly learning the solution operators of parametric reactive transport equations with various boundary conditions, multiple phases, and multi-physical fields through five examples. This approach offers significant computational savings, potentially reducing the computation time by 50–1000 times compared with the finite-element method. Our work may provide a robust alternative for solving parametric reactive transport equations in porous media, paving the way for exploring complex phenomena in porous media.

Characterizing Deep Learning Package Supply Chains in PyPI: Domains, Clusters, and Disengagement

Deep learning (DL) frameworks have become the cornerstone of the rapidly developing DL field. Through installation dependencies specified in the distribution metadata, numerous packages directly or transitively depend on DL frameworks, layer after layer, forming DL package supply chains (SCs), which are critical for DL frameworks to remain competitive. However, vital knowledge on how to nurture and sustain DL package SCs is still lacking. Achieving this knowledge may help DL frameworks formulate effective measures to strengthen their SCs to remain competitive and shed light on dependency issues and practices in the DL SC for researchers and practitioners. In this paper, we explore the domains, clusters, and disengagement of packages in two representative PyPI DL package SCs to bridge this knowledge gap. We analyze the metadata of nearly six million PyPI package distributions and construct version-sensitive SCs for two popular DL frameworks: TensorFlow and PyTorch. We find that popular packages (measured by the number of monthly downloads) in the two SCs cover 34 domains belonging to eight categories. Applications , Infrastructure , and Sciences categories account for over 85% of popular packages in either SC and TensorFlow and PyTorch SC have developed specializations on Infrastructure and Applications packages, respectively. We employ the Leiden community detection algorithm and detect 131 and 100 clusters in the two SCs. The clusters mainly exhibit four shapes: Arrow, Star, Tree, and Forest with increasing dependency complexity. Most clusters are Arrow or Star, while Tree and Forest clusters account for most packages (Tensorflow SC: 70.7%, PyTorch SC: 92.9%). We identify three groups of reasons why packages disengage from the SC (i.e., remove the DL framework and its dependents from their installation dependencies): dependency issues, functional improvements, and ease of installation. The most common reason in TensorFlow SC is dependency incompatibility and in PyTorch SC is to simplify functionalities and reduce installation size. Our study provides rich implications for DL framework vendors, researchers, and practitioners on the maintenance and dependency management practices of PyPI DL SCs.

Vehicle load identification based on bridge response using deep convolutional neural network

ABSTRACT Ensuring bridge safety and longevity through precise vehicle load characterization is crucial for maintaining structural integrity and efficiency. This research highlights the efficacy of deep convolutional neural networks (DCNNs) in analyzing bridge responses. The proposed methodology incorporates abridge model developed in ANSYS APDL, integrated subsequently with Universal Mechanism (UM) software for simulating vehicle-bridge interactions, considering factors such as vehicle load, speed, road surface roughness, and ambient noise. To validate the accuracy and robustness of the DCNN models, the study involved training and testing them on diverse datasets, each comprising 250 randomized samples for various parameters such as load, speed, and surface roughness. Two popular deep learning frameworks, ResNet and Inception models, were employed to assess their capability to identify these critical vehicle-bridge interaction characteristics accurately. The results reveal that the Inception model excelled in vehicle load identification, achieving an impressive validation accuracy of 99.9%. In contrast, ResNet outperformed in vehicle speed identification with a99% validation accuracy, surpassing Inception’s 97.9%. Both models exhibited remarkable resilience to noise, with road surface roughness identification demonstrating exceptional performance, including aperfect 100% training accuracy for ResNet and 99.9% for Inception. Furthermore, to validate the noise robustness of the DCNN model, the load identification is conducted under the effects of measurement noise, and the identification results show that the DCNN model can accurately identify vehicle loads with accuracies of 97.9% at 10 dB and 99.7% at 20 dB, respectively. These findings firmly establish the DCNNs’ capability to accurately identify vehicle load, speed, and road surface roughness, emphasizing their potential to significantly improve bridge safety and longevity.

Nonlinear Spiking Neural Systems With Autapses for Predicting Chaotic Time Series.

Spiking neural P (SNP) systems are a class of distributed and parallel neural-like computing models that are inspired by the mechanism of spiking neurons and are 3rd-generation neural networks. Chaotic time series forecasting is one of the most challenging problems for machine learning models. To address this challenge, we first propose a nonlinear version of SNP systems, called nonlinear SNP systems with autapses (NSNP-AU systems). In addition to the nonlinear consumption and generation of spikes, the NSNP-AU systems have three nonlinear gate functions, which are related to the states and outputs of the neurons. Inspired by the spiking mechanisms of NSNP-AU systems, we develop a recurrent-type prediction model for chaotic time series, called the NSNP-AU model. As a new variant of recurrent neural networks (RNNs), the NSNP-AU model is implemented in a popular deep learning framework. Four datasets of chaotic time series are investigated using the proposed NSNP-AU model, five state-of-the-art models, and 28 baseline prediction models. The experimental results demonstrate the advantage of the proposed NSNP-AU model for chaotic time series forecasting.

Toward Accurate Binarized Neural Networks With Sparsity for Mobile Application.

While binarized neural networks (BNNs) have attracted great interest, popular approaches proposed so far mainly exploit the symmetric sign function for feature binarization, i.e., to binarize activations into -1 and +1 with a fixed threshold of 0. However, whether this option is optimal has been largely overlooked. In this work, we propose the Sparsity-inducing BNN (Si-BNN) to quantize the activations to be either 0 or +1, which better approximates ReLU using 1-bit. We further introduce trainable thresholds into the backward function of binarization to guide the gradient propagation. Our method dramatically outperforms the current state-of-the-art, lowering the performance gap between full-precision networks and BNNs on mainstream architectures, achieving the new state-of-the-art on binarized AlexNet (Top-1 50.5%), ResNet-18 (Top-1 62.2%), and ResNet-50 (Top-1 68.3%). At inference time, Si-BNN still enjoys the high efficiency of bit-wise operations. In our implementation, the running time of binary AlexNet on the CPU can be competitive with the popular GPU-based deep learning framework.

Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models.

In deep learning, different kinds of deep networks typically need different optimizers, which have to be chosen after multiple trials, making the training process inefficient. To relieve this issue and consistently improve the model training speed across deep networks, we propose the ADAptive Nesterov momentum algorithm, Adan for short. Adan first reformulates the vanilla Nesterov acceleration to develop a new Nesterov momentum estimation (NME) method, which avoids the extra overhead of computing gradient at the extrapolation point. Then Adan adopts NME to estimate the gradient's first- and second-order moments in adaptive gradient algorithms for convergence acceleration. Besides, we prove that Adan finds an ϵ-approximate first-order stationary point within O(ϵ-3.5) stochastic gradient complexity on the non-convex stochastic problems (e.g.deep learning problems), matching the best-known lower bound. Extensive experimental results show that Adan consistently surpasses the corresponding SoTA optimizers on vision, language, and RL tasks and sets new SoTAs for many popular networks and frameworks, egResNet, ConvNext, ViT, Swin, MAE, DETR, GPT-2, Transformer-XL, and BERT. More surprisingly, Adan can use half of the training cost (epochs) of SoTA optimizers to achieve higher or comparable performance on ViT, GPT-2, MAE, etc, and also shows great tolerance to a large range of minibatch size, e.g.from 1k to 32k. Code is released at https://github.com/sail-sg/Adan, and has been used in multiple popular deep learning frameworks or projects.

Rise of Distributed Deep Learning Training in the Big Model Era: From a Software Engineering Perspective

Deep learning (DL) has become a key component of modern software. In the “ big model ” era, the rich features of DL-based software (i.e., DL software) substantially rely on powerful DL models, e.g., BERT, GPT-3, and the recently emerging GPT-4, which are trained on the powerful cloud with large datasets. Hence, training effective DL models has become a vital stage in the whole software lifecycle. When training deep learning models, especially those big models, developers need to parallelize and distribute the computation and memory resources amongst multiple devices (e.g., a cluster of GPUs) in the training process, which is known as distributed deep learning training , or distributed training for short. However, the unique challenges that developers encounter in distributed training process have not been studied in the software engineering community. Given the increasingly heavy dependence of current DL-based software on distributed training, this paper aims to fill in the knowledge gap and presents the first comprehensive study on developers’ issues in distributed training. To this end, we focus on popular DL frameworks that support distributed training (including TensorFlow, PyTorch, Keras, and Horovod) and analyze 1,131 real-world developers’ issues about using these frameworks reported on Stack Overflow and GitHub. We construct a fine-grained taxonomy consisting of 30 categories regarding the fault symptoms and summarize common fix patterns for different symptoms. We find that: (1) many distributed-specific faults and non-distributed-specific faults inherently share the same fault symptoms, making it challenging to debug; (2) most of the fault symptoms have frequent fix patterns; (3) about half of the faults are related to system-level configurations. Based on the results, we suggest actionable implications on research avenues that can potentially facilitate the distributed training to develop DL-based software, such as focusing on the frequent and common fix patterns when designing testing or debugging tools, developing efficient testing and debugging techniques for communication configuration along with the synthesis of network configuration analysis, designing new multi-device checkpoint-and-replay techniques to help reproduction, and designing serverless APIs for cloud platforms.

Automated Video-Based Capture of Crustacean Fisheries Data Using Low-Power Hardware.

This work investigates the application of Computer Vision to the problem of the automated counting and measuring of crabs and lobsters onboard fishing boats. The aim is to provide catch count and measurement data for these key commercial crustacean species. This can provide vital input data for stock assessment models, to enable the sustainable management of these species. The hardware system is required to be low-cost, have low-power usage, be waterproof, available (given current chip shortages), and able to avoid over-heating. The selected hardware is based on a Raspberry Pi 3A+ contained in a custom waterproof housing. This hardware places challenging limitations on the options for processing the incoming video, with many popular deep learning frameworks (even light-weight versions) unable to load or run given the limited computational resources. The problem can be broken into several steps: (1) Identifying the portions of the video that contain each individual animal; (2) Selecting a set of representative frames for each animal, e.g, lobsters must be viewed from the top and underside; (3) Detecting the animal within the frame so that the image can be cropped to the region of interest; (4) Detecting keypoints on each animal; and (5) Inferring measurements from the keypoint data. In this work, we develop a pipeline that addresses these steps, including a key novel solution to frame selection in video streams that uses classification, temporal segmentation, smoothing techniques and frame quality estimation. The developed pipeline is able to operate on the target low-power hardware and the experiments show that, given sufficient training data, reasonable performance is achieved.

Challenges, Corresponding Solutions, and Applications of Generative Adversarial Networks

Generative adversarial networks (GANs) have become a popular deep learning framework in the field of artificial intelligence. Researchers have developed various types of GANs, such as Conditional GANs (CGANs), Mode Dropping GANs (MDGANs), and Wasserstein GANs (WGANs), which have been applied to many different fields. Despite the success of GANs in various tasks, there are still some challenges that need to be addressed. For example, model collapse, non-convergence, and diminished gradient could hinder the training of GANs. In this paper, the authors first introduce the basics of GANs before highlighting a few common problems associated with GANs and their corresponding solutions. For instance, they discuss techniques such as mini-batch discrimination and batch normalization that can help mitigate model collapse and diminish gradient problems. Additionally, they cover methods that can improve the convergence rate and stability of GANs, such as using alternative loss functions and incorporating regularization algorithms. Finally, this paper briefly introduces some recent applications of GANs in image generation, video prediction, and other areas. Despite the challenges that still exist, GANs have shown great promise in various fields, and with further research, GANs have the potential to become more robust and efficient models for generating high-quality synthetic data.

A complementary integrated Transformer network for hyperspectral image classification

AbstractIn the past, convolutional neural network (CNN) has become one of the most popular deep learning frameworks, and has been widely used in Hyperspectral image classification tasks. Convolution (Conv) in CNN uses filter weights to extract features in local receiving domain, and the weight parameters are shared globally, which more focus on the high‐frequency information of the image. Different from Conv, Transformer can obtain the long‐term dependence between long‐distance features through modelling, and adaptively focus on different regions. In addition, Transformer is considered as a low‐pass filter, which more focuses on the low‐frequency information of the image. Considering the complementary characteristics of Conv and Transformer, the two modes can be integrated for full feature extraction. In addition, the most important image features correspond to the discrimination region, while the secondary image features represent important but easily ignored regions, which are also conducive to the classification of HSIs. In this study, a complementary integrated Transformer network (CITNet) for hyperspectral image classification is proposed. Firstly, three‐dimensional convolution (Conv3D) and two‐dimensional convolution (Conv2D) are utilised to extract the shallow semantic information of the image. In order to enhance the secondary features, a channel Gaussian modulation attention module is proposed, which is embedded between Conv3D and Conv2D. This module can not only enhance secondary features, but suppress the most important and least important features. Then, considering the different and complementary characteristics of Conv and Transformer, a complementary integrated Transformer module is designed. Finally, through a large number of experiments, this study evaluates the classification performance of CITNet and several state‐of‐the‐art networks on five common datasets. The experimental results show that compared with these classification networks, CITNet can provide better classification performance.

Optimizing LSTM Models for EUR/USD Prediction in the context of reducing energy consumption: An Analysis of Mean Squared Error, Mean Absolute Error and R-Squared

The purpose of this study was to develop and evaluate a Long Short-Term Memory (LSTM) model for Forex prediction. The data used was reprocessed and the LSTM model was developed and trained using a supervised learning approach with popular deep learning frameworks. The performance of the model was evaluated using metrics such as Mean Squared Error (MSE), Mean Absolute Error (MAE), and R-squared. In addition, we examined the literature on energy efficiency, highlighting its potential for reducing computational load and, consequently, energy consumption. We also considered the environmental impact of using such models. The results showed that the LSTM model was effective in Forex prediction and demonstrated superior performance compared to other predictive models. The best model among the several LSTM models evaluated had 90 epochs. These results provide evidence for the efficacy of the LSTM model in Forex prediction and highlight the potential benefits of using deep learning techniques in this field, particularly in terms of energy efficiency and environmental sustainability.

Marigold Flower Blooming Stage Detection in Complex Scene Environment using Faster RCNN with Data Augmentation

In recent years, flower growing has developed into a lucrative agricultural sector that provides employment and business opportunities for small and marginal growers in both urban and rural locations in India. One of the most often cultivated flowers for landscaping design is the Marigold flower. It is also widely used to create garlands for ceremonial and social occasions using loose flowers. Understanding the appropriate stage of harvesting for each plant species is essential to ensuring the quality of the flowers after they have been picked. It has been demonstrated that human assessors consistently used a category scoring system to evaluate various flowering stages. Deep learning and convolutional neural networks have the potential to revolutionize agriculture by enabling efficient analysis of large-scale data. In order to address the problem of Marigold flower stages detection and classification in complex real-time field scenarios, this study proposes a fine-tuned Faster RCNN with ResNet50 network coupled with data augmentation. Faster RCNN is a popular deep learning framework for object detection that uses a region proposal network to efficiently identify object locations and features in an image. The Marigold flower dataset was collected from three different Marigold fields in the Anand District of Gujarat State, India. The collection includes of photos that were taken outdoors in natural light at various heights, angles, and distances. We have developed and fine-tuned a Faster RCNN detection and classification model to be particularly sensitive to Marigold flowers, and we have compared the generated method's performance to that of other cutting-edge models to determine its accuracy and effectiveness.

Thermal Behavior Modeling Based on BP Neural Network in Keras Framework for Motorized Machine Tool Spindles.

This paper presents the development and evaluation of neural network models using a small input-output dataset to predict the thermal behavior of a high-speed motorized spindles. Different neural multi-output regression models were developed and evaluated using Keras, one of the most popular deep learning frameworks at the moment. ANN was developed and evaluated considering the following: the influence of the topology (number of hidden layers and neurons within), the learning parameter, and validation techniques. The neural network was simulated using a dataset that was completely unknown to the network. The ANN model was used for analyzing the effect of working conditions on the thermal behavior of the motorized grinder spindle. The prediction accuracy of the ANN model for the spindle thermal behavior ranged from 95% to 98%. The results show that the ANN model with small datasets can accurately predict the temperature of the spindle under different working conditions. In addition, the analysis showed a very strong effect of type coolant on spindle unit temperature, particularly for intensive cooling with water.

A comprehensive empirical study on bug characteristics of deep learning frameworks

Deep Learning (DL) frameworks enable developers to build DNN models without learning the underlying algorithms and models. While some of these DL-based software systems have been deployed in safety-critical areas, such as self-driving cars and medical diagnostics, for DL frameworks, characterizing their bugs and thus helping researchers to design specific quality assurance techniques become desperately needed. Our research aims to characterize bugs typical of DL frameworks at the source code level for an in-depth analysis of bug symptoms, root causes, and bug fixes. In this way, we hope to provide insights for researchers to design automatic quality assurance techniques, such as automatic repair techniques and fault location techniques, applicable to DL frameworks and DL-based software systems. We started by summarizing the DL framework reference architecture and proposing the DL framework bug taxonomy. Then, we mined 1,127 DL framework bug reports from eight popular DL frameworks and labeled the bug types, root causes, and symptoms. Finally, we discussed the bug characteristics and explored how developers could possibly deal with these bugs. Our main findings are: (i) DNN model building bugs and general type bugs accounted for one-third of the total defects. (ii) DNN model building bugs are more prone to algorithm logic constraints, internal API errors, and data/numerical errors. (iii) Fifteen bug-fixing patterns are summarized, providing reference for common DL framework bug repair and future research on the development of automatic DL framework bug detection tools. By analyzing the bug-fixing changes, we characterize the occurrences, root causes, symptoms, and fixing of these bugs. The study results have provided researchers with insights into how to ensure DL framework quality and presented actionable suggestions for DL framework developers to improve their code quality.

DMA-Net: DeepLab With Multi-Scale Attention for Pavement Crack Segmentation

Cracks are important indicators of pavement structural and operational conditions. Early pavement crack detection and treatments can help extend pavement service life, reduce fuel consumption, and improve safety and ride quality. Pavement distress surveys have traditionally been performed manually by visually inspecting the roads, which is labor-intensive and time-consuming. Therefore, computer-vision-based automated crack detection has great practical significance in pavement maintenance and traffic safety. Traditional image processing techniques are sensitive to noise in images and are thus likely to miss detecting some cracks due to the crack texture variety, complex lighting conditions, and various similar but irrelevant objects on the road. This paper adopts and enhances DeepLabv3+, a popular deep learning framework for semantic image segmentation, for road pavement crack detection. We propose a multi-scale attention module in the decoder of DeepLabv3+ to generate an attention mask and dynamically assign weights between high-level and low-level feature maps. Compared with fixed weights across different features, the dynamic weights strategy can assign more reasonable weights to different feature maps. Ablation experiments show that the attention mask can effectively help the model better combine multi-scale features and generate more accurate pavement crack segmentation results. The proposed method achieves state-of-the-art results on three benchmarks, including Crack500, DeepCrack, and FMA (Fitchburg Municipal Airport) datasets. We further test it on pavement crack images captured by smartphones, and the results show that it provides a viable approach to road pavement crack segmentation in practice with excellent performance.

"Mitigation and Identification of Camouflage Attack Over Computer Vision Applications"

Computer vision technologies are now commonly used in real-time image and video recognition applications using deep neural networks. Scaling or Mitigation is the basic input pre-processing feature in these implementations. Image scaling methods are designed to maintain visual elements before and after scaling, and are widely utilized in a variety of visual and image processing applications. Content disguising attacks may be carried out by taking advantage of the picture scaling mechanism, which causes the machine's extracted content to be drastically different from the input before scaling. In this project the actual input image with dimension Am×n is resized to an attacked image dimension Om1×n1, which will mislead the classifier as the input is attacked image. To illustrate the threats from such camouflage attacks, some computer vision applications taken as targeted victims that includes multiple image classification applications based on popular deep learning frameworks like OpenCV, pillow. This attack is not detected while checking with the YOLOV4 demo. To defend against such attacks, in this project using Dhash i.e., difference hash concept. The hash the value of the actual image and the resized are same when using the Dhash. If the image is camouflaged the means the content is modified so the Dhash value of the image also changed if the image is not camouflaged the then the hash value of the actual image and resized images are same. By using the Dhash concept detecting the image is camouflaged or not.

NURBS-Diff: A Differentiable Programming Module for NURBS

Boundary representations (B-reps) using Non-Uniform Rational B-splines (NURBS) are the de facto standard used in CAD, but their utility in deep learning-based approaches is not well researched. We propose a differentiable NURBS module to integrate NURBS representations of CAD models with deep learning methods. We mathematically define the derivatives of the NURBS curves or surfaces with respect to the input parameters (control points, weights, and the knot vector). These derivatives are used to define an approximate Jacobian used for performing the “backward” evaluation to train the deep learning models. We have implemented our NURBS module using GPU-accelerated algorithms and integrated it with PyTorch, a popular deep learning framework. We demonstrate the efficacy of our NURBS module in performing CAD operations such as curve or surface fitting and surface offsetting. Further, we show its utility in deep learning for unsupervised point cloud reconstruction and enforce analysis constraints. These examples show that our module performs better for certain deep learning frameworks and can be directly integrated with any deep-learning framework requiring NURBS.

A Shallow-to-Deep Feature Fusion Network for VHR Remote Sensing Image Classification

With more detailed spatial information being represented in very-high-resolution (VHR) remote sensing images, stringent requirements are imposed on accurate image classification. Due to the diverse land-objects with intraclass variation and interclass similarity, efficient and fine classification of VHR images especially in complex scenes is challenging. Even for some popular deep learning (DL) frameworks, geometric details of land-object may be lost in deep feature levels, so it is difficult to maintain the highly-detailed spatial information (e.g., edges, small objects) only relying on the last high-level layer. Moreover, many of the newly developed DL methods require massive well-labeled samples, which inevitably deteriorates the model generalization ability under the few-shot learning. Therefore, in this paper, a lightweight shallow-to-deep feature fusion network (SDF2N) is proposed for VHR image classification, where the traditional machine learning (ML) and DL schemes are integrated to learn rich and representative information to improve the classification accuracy. In particular, the shallow spectral-spatial features are first extracted, and then a novel triple-stage fusion (TSF) module is designed to learn the saliency and discriminative information at different levels for classification. The TSF module includes three feature fusion stages, i.e., low-level spectral-spatial feature fusion, middle-level multi-scale feature fusion, and high-level multi-layer feature fusion. The proposed SDF2N takes advantages of the shallow-to-deep features, which can extract representative and complementary information of crossing layers. It is important to note that even with limited training samples, the SDF2N still can achieve satisfying classification performance. Experimental results obtained on three real VHR remote sensing data sets including two multispectral and one airborne hyperspectral images covering complex urban scenarios confirm the effectiveness of the proposed approach compared with the state-of-the-art methods.