Spatial Feature Fusion Research Articles

Enhanced spectral attention and adaptive spatial learning guided network for hyperspectral and LiDAR classification

Although the data fusion of hyperspectral images (HSI) and light detection and ranging (LiDAR) has provided significant gains for land-cover classification, it also brings technical obstacles (i.e., it is difficult to capture discriminative local and global spatial-spectral from redundant data and build interactions between heterogeneous data). In this paper, a classification network named enhanced spectral attention and adaptive spatial learning guided network (ESASNet) is proposed for the joint use of HSI and LiDAR. Specifically, first, by combining a convolutional neural network (CNN) with the transformer, adaptive spatial learning (ASL) and enhanced spectral learning (ESL) are proposed to learn the spectral-spatial features from the HSI data and the elevation features from the LiDAR data in the local and global receptive field. Second, considering the characteristics of HSI with a continuous, narrowband spectrum, ESL is designed by adding enhanced local self-attention to enhance the mining of the spectral correlations across the adjacent spectrum. Finally, a feature fusion module is proposed to ensure an efficient information exchange between HSI and LiDAR during spectral features and spatial feature fusion. Experimental evaluations on the HSI-LiDAR dataset clearly illustrate that ESASNet performs better in feature extraction than the state-of-the-art methods. The code is available at https://github.com/AirsterMode/ESASNet.

Underwater Engineering Crack Identification based on Lightweight Convolutional Neural Network

Convolutional neural network is an effective model for image feature recognition, which is widely used in crack recognition and risk detection. The crack identification of underwater engineering involves a huge scale of data, and the traditional R-CNN model and Faster R-CNN model have the problems of large number of model parameters and low efficiency for crack identification. Therefore, a lightweight convolutional neural network based crack identification method for underwater engineering is proposed to optimize the efficiency and accuracy of crack identification. Deep separable convolution is used to replace the common convolution of YOLOX network (an open source high-performance object detection algorithm), reducing the redundant parameters of the model to achieve lightweight design. Following the backbone feature extraction network framework of YOLOX, Transformer (a structure based on attention mechanism and forward neural network) vision module is adopted to replace the CSP structure (a feature extraction structure) at the end of the backbone network, adding attention mechanism and enhancing key information extraction capability. Make up for the problem of decreasing feature extraction accuracy caused by decreasing parameter number; Finally, the adaptive spatial feature fusion strategy is used instead of the traditional feature pyramid cascade to implement feature fusion and improve the feature extraction ability of small targets. The test results show that the improved YOLOX lightweight convolutional neural network has good prediction ability for underwater engineering cracks, and can accurately select the crack location, with fewer false detection and missed detection cases. Ablation experiments show that this study builds a lightweight convolutional neural network model, which improves the accuracy of the model to identify underwater engineering cracks. The improved lightweight convolutional neural network is suitable for crack identification in underwater engineering and can be widely applied in the field of underwater engineering risk identification.

Domain Alignment Dynamic Spectral and Spatial Feature Fusion for Hyperspectral Change Detection

Domain Alignment Dynamic Spectral and Spatial Feature Fusion for Hyperspectral Change Detection

Pine wilt disease detection algorithm based on improved YOLOv5.

Pine wilt disease (PWD) poses a significant threat to forests due to its high infectivity and lethality. The absence of an effective treatment underscores the importance of timely detection and isolation of infected trees for effective prevention and control. While deep learning techniques combined unmanned aerial vehicle (UAV) remote sensing images offer promise for accurate identification of diseased pine trees in their natural environments, they often demand extensive prior professional knowledge and struggle with efficiency. This paper proposes a detection model YOLOv5L-s-SimAM-ASFF, which achieves remarkable precision, maintains a lightweight structure, and facilitates real-time detection of diseased pine trees in UAV RGB images under natural conditions. This is achieved through the integration of the ShuffleNetV2 network, a simple parameter-free attention module known as SimAM, and adaptively spatial feature fusion (ASFF). The model boasts a mean average precision (mAP) of 95.64% and a recall rate of 91.28% in detecting pine wilt diseased trees, while operating at an impressive 95.70 frames per second (FPS). Furthermore, it significantly reduces model size and parameter count compared to the original YOLOv5-Lite. These findings indicate that the proposed model YOLOv5L-s-SimAM-ASFF is most suitable for real-time, high-accuracy, and lightweight detection of PWD-infected trees. This capability is crucial for precise localization and quantification of infected trees, thereby providing valuable guidance for effective management and eradication efforts.

A lightweight network based on local–global feature fusion for real-time industrial invisible gas detection with infrared thermography

The detection of industrial invisible gas plays a vital role in preventing environmental pollution and fire accidents. Optical gas imaging (OGI) with infrared thermography is widely used in the field of gas leak monitoring and treatment by visualizing gases to quickly and accurately detect and locate the gas leak sources. However, this method still relies on manual visual inspection. Existing automatic visual gas detection methods suffer from insufficient gas feature extraction and high computational cost due to the indistinct gas features in thermal images. To address these problems, we propose a new lightweight network specialized for thermal gas feature extraction, namely GasViT, sufficiently extracting gas features at very low computational cost by local–global feature fusion. Specifically, two new feature extraction modules Multi-scale Fusion Feature Attention (MsFFA) and Multi-head Linear Self-attention (MhLSa) are proposed for GasViT. MsFFA enhances the gas local feature extraction ability by constructing multi-scale channel and spatial feature fusion maps, enabling the network to focus on more valid local information. MhLSa complements the gas global features with very low computational cost by efficiently encoding the global information of the image in terms of the innovative linear self-attention mechanism. Our experimental results on the self-made Industrial Invisible Gas (IIG) Dataset show GasViT achieves 82.7% mAP50, significantly outperforming the state-of-the-art lightweight networks. Moreover, GasViT achieves 33 FPS real-time detection with a running memory footprint of only 47.2 MB on edge computing devices, making it extremely suitable for portable and embedded detection devices than existing methods to cover gas leakage detection in complex and hazardous industrial scenarios.

MSSFF: Advancing Hyperspectral Classification through Higher-Accuracy Multistage Spectral–Spatial Feature Fusion

This paper presents the MSSFF (multistage spectral–spatial feature fusion) framework, which introduces a novel approach for semantic segmentation from hyperspectral imagery (HSI). The framework aims to simplify the modeling of spectral relationships in HSI sequences and unify the architecture for semantic segmentation of HSIs. It incorporates a spectral–spatial feature fusion module and a multi-attention mechanism to efficiently extract hyperspectral features. The MSSFF framework reevaluates the potential impact of spectral and spatial features on segmentation models and leverages the spectral–spatial fusion module (SSFM) in the encoder component to effectively extract and enhance these features. Additionally, an efficient Transformer (ET) is introduced in the skip connection part of deep features to capture long-term dependent features and extract global spectral–spatial information from the entire feature map. This highlights the significant potential of Transformers in modeling spectral–spatial feature maps within the context of hyperspectral remote sensing. Moreover, a spatial attention mechanism is adopted in the shallow skip connection part to extract local features. The framework demonstrates promising capabilities in hyperspectral remote sensing applications. The conducted experiments provide valuable insights for optimizing the model depth and the order of feature fusion, thereby contributing to the advancement of hyperspectral semantic segmentation research.

Anchor boxes adaptive optimization algorithm for maritime object detection in video surveillance

With the development of the marine economy, video surveillance has become an important technical guarantee in the fields of marine engineering, marine public safety, marine supervision, and maritime traffic safety. In video surveillance, maritime object detection (MOD) is one of the most important core technologies. Affected by the size of maritime objects, distance, day and night weather, and changes in sea conditions, MOD faces challenges such as false detection, missed detection, slow detection speed, and low accuracy. However, the existing object detection algorithms usually adopt predefined anchor boxes to search and locate for objects of interest, making it difficult to adapt to maritime objects’ complex features, including the varying scale and large aspect ratio difference. Therefore, this paper proposes a maritime object detection algorithm based on the improved convolutional neural network (CNN). Firstly, a differential-evolutionary-based K-means (DK-means) anchor box clustering algorithm is proposed to obtain adaptive anchor boxes to satisfy the maritime object characteristics. Secondly, an adaptive spatial feature fusion (ASFF) module is added in the neck network to enhance multi-scale feature fusion. Finally, focal loss and efficient intersection over union (IoU) loss are adopted to replace the original loss function to improve the network convergence speed. The experimental results on the Singapore maritime dataset show that our proposed algorithm improves the average precision by 7.1%, achieving 72.7%, with a detection speed of 113 frames per second, compared with You Only Look Once v5 small (YOLOv5s). Moreover, compared to other counterparts, it can achieve a better speed–accuracy balance, which is superior and feasible for the complex maritime environment.

Facial Wrinkle Detection with Multiscale Spatial Feature Fusion Based on Image Enhancement and ASFF-SEUnet

Wrinkles, crucial for age estimation and skin quality assessment, present challenges due to their uneven distribution, varying scale, and sensitivity to factors like lighting. To overcome these challenges, this study presents facial wrinkle detection with multiscale spatial feature fusion based on image enhancement and an adaptively spatial feature fusion squeeze-and-excitation Unet network (ASFF-SEUnet) model. Firstly, in order to improve wrinkle features and address the issue of uneven illumination in wrinkle images, an innovative image enhancement algorithm named Coiflet wavelet transform Donoho threshold and improved Retinex (CT-DIR) is proposed. Secondly, the ASFF-SEUnet model is designed to enhance the accuracy of full-face wrinkle detection across all age groups under the influence of lighting factors. It replaces the encoder part of the Unet network with EfficientNet, enabling the simultaneous adjustment of depth, width, and resolution for improved wrinkle feature extraction. The squeeze-and-excitation (SE) attention mechanism is introduced to grasp the correlation and importance among features, thereby enhancing the extraction of local wrinkle details. Finally, the adaptively spatial feature fusion (ASFF) module is incorporated to adaptively fuse multiscale features, capturing facial wrinkle information comprehensively. Experimentally, the method excels in detecting facial wrinkles amid complex backgrounds, robustly supporting facial skin quality diagnosis and age assessment.

A novel feature enhancement and semantic segmentation scheme for identifying low-contrast ocean oil spills

The oil spill accidents on the sea surface pose a severe threat to the marine environment and human health. This paper proposes a novel Semantic Segmentation Network (SSN) for processing oil spill images so that low-contrast oil spills on the sea surface can be accurately identified. After the detection accuracy and real-time performance of the current SSNs are compared, the basic network architecture of DeeplabV3+ based target detection is analyzed. The standard convolution is replaced by the Omni-dimensional Dynamic Convolution (ODConv) in the Ghost Module Depth-Wise separable Convolution (DWConv) to further enhance the feature extraction ability of the network. Furthermore, a new DeeplabV3+ based network with ODGhostNetV2 is constructed as the main feature extraction module, and an Adaptive Triplet Attention (ATA) module is deployed in the encoder and decoder at the same time. This not only improves the richness of semantic features but also increases the following receptive fields of the network model. ATA integrates the Adaptively Spatial Feature Fusion (ASFF) module to optimize the weight assignment problem in the feature map fusion process. The ablation experiments are conducted to verify the proposed network which show high accuracy and good real-time performance for the oil spill detection.

DFENet: Double Feature Enhanced Class Agnostic Counting Methods

Object counting is a basic computer vision task, which can estimate the number of each object in an image, thus providing valuable information. In dense scenes, there are huge differences in target individual scale, and the different target individual scale leads to low accuracy of target count. In addition, most of the existing target count datasets in the field require a lot of manual creation and annotation, which increases the cost and difficulty of the dataset, lack of ease of use and portability. To solve these problems, this paper proposes a class agnostic counting method Double Feature Enhancement Net based on improved Bilinear Matching Network+ (BMNet+). By introducing the feature enhancement module based on the principle of conditional random field and the adaptively spatial feature fusion module, combined with the feature similarity measurement strategy of bilinear matching network, the method can effectively extract the target features of different scales, enhance the adaptability to the targets with large scale changes, and improve the counting performance of the network. Experiments were carried out on FSC-147 data set, and the experimental results show that the proposed model has been further improved in counting accuracy. The MAE and MSE of the verification set are 15.03 and 54.53 respectively. In the test set, MAE reaches 13.65, MSE reaches 89.54, and the counting performance is at the advanced level in the field.

Vehicle classification based on audio-visual feature fusion with low-quality images and noise

In Intelligent Transport Systems (ITS), vision is the primary mode of perception. However, vehicle images captured by low-cost traffic cameras under challenging weather conditions often suffer from poor resolution and insufficient detail representation. On the other hand, vehicle noise provides complementary auditory features that offer advantages such as environmental adaptability and a large recognition distance. To address these limitations and enhance the accuracy of low-quality traffic surveillance classification and identification, an effective audio-visual feature fusion method is crucial. This paper presents a research study that establishes an Urban Road Vehicle Audio-visual (URVAV) dataset specifically designed for low-quality images and noise recorded in complex weather conditions. For low-quality vehicle image classification, the paper proposes a simple Convolutional Neural Network (CNN)-based model called Low-quality Vehicle Images Net (LVINet). Additionally, to further enhance classification accuracy, a spatial channel attention-based audio-visual feature fusion method is introduced. This method converts one-dimensional acoustic features into a two-dimensional audio Mel-spectrogram, allowing for the fusion of auditory and visual features. By leveraging the high correlation between these features, the representation of vehicle characteristics is effectively enhanced. Experimental results demonstrate that LVINet achieves a classification accuracy of 93.62% with reduced parameter count compared to existing CNN models. Furthermore, the proposed audio-visual feature fusion method improves classification accuracy by 7.02% and 4.33% when compared to using single audio or visual features alone, respectively.

LF-YOLOv4: a lightweight detection model for enhancing the fusion of image features of surface defects in lithium batteries

In response to the low accuracy of traditional methods for detecting surface defects in lithium batteries, as well as the problems of large model size and high computational complexity in current detection models, this article proposes a new lightweight LF-YOLOv4 model that enhances image feature fusion. Firstly, replace the CSPDarknet53 backbone network in YOLOv4 with a lightweight MobileNetv2 network, thereby greatly reducing the computational parameters of the network while ensuring the ability to extract features. Secondly, in order to further reduce the number of model parameters and computational complexity, and minimize potential accuracy loss as much as possible, an improved depthwise separable convolution (DSC-SE-HsId) in this article was studied, which replaced some ordinary convolutions in the Neck and Head networks. Finally, in order to compensate for the partial accuracy loss caused by lightweight operations, and also to fuse feature maps of different scales to obtain more complete feature information, a new lightweight adaptive spatial feature fusion module (LSE-ASFF) in this article was studied and embedded behind the existing path aggregation network (PANet). In order to verify the performance and widespread applicability of the improved model, we conducted tests using the self-built lithium battery surface defect dataset, and the steel surface defect dataset provided by Northeastern University. Moreover, in order to prevent the network training from over fitting due to too few defect samples in the self-built dataset, we also used image enhancement to expand the dataset. And the experiment results show that the improved model proposed in this article achieves the highest TOPSIS score in both experimental datasets. Among them, compared with YOLOv4 on the self-built dataset, our improved model not only increases mAP50 by 2.97%, reaching 97.83%, but also has model parameters of only 18.16% of the original model, floating point of operations of only 13.87% of the original model, model size of only 21.02% of the original model, and model training time shortened by 30.67% compared to the original model. At last, the effectiveness and superiority of the improved model in this article are demonstrated through example analysis and comparison.

Textile defect detection based on multi‐proportion spatial attention mechanism and channel memory feature fusion network

AbstractThis paper presents a textile defect detection method that utilizes a multi‐proportion spatial attention mechanism and channel memory feature fusion network by addressing the difficulties presented by complicated shapes and large size variations. In particular, a multi‐proportion spatial attention mechanism (MPAM) is introduced, which employs multi‐proportion convolution to improve the backbone network's capacity to detect non‐uniform structural defects. Additionally, the generality and adaptability of the model are enhanced by a multi‐scale spatial pyramid pooling structure (MS‐SPP). Second, a channel attention mechanism‐based memory feature fusion network is developed, which incorporates channel attention to adaptive weight the feature channels, focusing on crucial information channels to efficiently fuse contextual features and enhance the model's memory capacity. Finally, a novel efficient Wise‐IoU (EWIoU) loss function is proposed, which utilizes a dynamic non‐monotonic focusing mechanism to increase the penalty on distance measurement, thus enhancing the model's detection performance. Experiment findings on the ZJU‐Leaper and Tianchi textile datasets reveal that compared to the YOLOv7 baseline, the method in this paper has an increase of 6.5 and 2 percentage points, respectively, and the detection accuracy is better than most existing networks.

Spatio-Temporal Alignment and Track-To-Velocity Module for Tropical Cyclone Forecast

The prediction of a tropical cyclone’s trajectory is crucial for ensuring marine safety and promoting economic growth. Previous approaches to this task have been broadly categorized as either numerical or statistical methods, with the former being computationally expensive. Among the latter, multilayer perceptron (MLP)-based methods have been found to be simple but lacking in time series capabilities, while recurrent neural network (RNN)-based methods excel at processing time series data but do not integrate external information. Recent works have attempted to enhance prediction performance by simultaneously utilizing both time series and meteorological field data through feature fusion. However, these approaches have relatively simplistic methods for data fusion and do not fully explore the correlations between different modalities. To address these limitations, we propose a systematic solution called TC-TrajGRU for predicting tropical cyclone tracks. Our approach improves upon existing methods in two main ways. Firstly, we introduce a Spatial Alignment Feature Fusion (SAFF) module to address feature misalignment issues in different dimensions. Secondly, our Track-to-Velocity (T2V) module leverages time series differences to integrate external information. Our experiments demonstrate that our approach yields highly accurate predictions comparable to the official optimal forecast for a 12 h period.

Research on Black Smoke Detection and Class Evaluation Method for Ships Based on YOLOv5s-CMBI Multi-Feature Fusion

To enhance the real-time detection accuracy of ship exhaust plumes and further quantify the degree of darkness, this study proposes a multi-feature fusion approach that combines the YOLOv5s-CMBI algorithm for ship exhaust plume detection with the Ringerman Blackness-based grading method. Firstly, diverse datasets are integrated and a subset of the data is subjected to standard optical model aerosolization to form a dataset for ship exhaust plume detection. Subsequently, building upon the YOLOv5s architecture, the CBAM convolutional attention mechanism is incorporated to augment the network’s focus on ship exhaust plume regions while suppressing irrelevant information. Simultaneously, inspired by the BiFPN structure with weighted bidirectional feature pyramids, a lightweight network named Tiny-BiFPN is devised to enable multi-path feature fusion. The Adaptive Spatial Feature Fusion (ASFF) mechanism is introduced to counteract the impact of feature scale disparities. The EIoU_Loss is employed as the localization loss function to enhance both regression accuracy and convergence speed of the model. Lastly, leveraging the k-means clustering algorithm, color information is mined through histogram analysis to determine clustering centers. The Mahalanobis distance is used to compute sample similarity, and the Ringerman Blackness-based method is employed to categorize darkness levels. Ship exhaust plume grades are estimated by computing a weighted average grayscale ratio between the effective exhaust plume region and the background region. Experimental results reveal that the proposed algorithm achieves improvements of approximately 3.8% in detection accuracy, 5.7% in recall rate, and 4.6% in mean average precision (mAP0.5) compared to the original model. The accuracy of ship exhaust plume darkness grading attains 92.1%. The methodology presented in this study holds significant implications for the establishment and application of future ship exhaust plume monitoring mechanisms.

CenterPoint-SE: A Single-Stage Anchor-Free 3-D Object Detection Algorithm With Spatial Awareness Enhancement

Real-time and accurate 3-D object detection is one of the foundational technologies for environmental perception in autonomous vehicles. However, the existing second-stage anchor-based 3-D object detection algorithms have high accuracy, but they are challenging in terms of computation complexity and latency. Due to poor perception of spatial features, the accuracy of the existing single-stage anchor-free detection algorithms with low latency are difficult to be implemented into autonomous vehicles. Therefore, we focus on enhancing the spatial perception ability of the anchor-free detection network based on CenterPoints. In this paper, we propose a single-stage anchor-free 3-D object detector CenterPoint-Space-Enhancement (CenterPoint-SE) algorithm and construct an efficient 3-D backbone network to extract fine-grained spatial geometric features by introducing a spatial attention mechanism and residual structure. At the same time, a powerful spatial semantic feature fusion module, the enhancement of feature fusion (EF-Fusion), is designed. In addition, we add a lightweight IoU prediction branch to improve the algorithm’s perception of various object sizes. Finally, we add a foreground point segmentation auxiliary training branch to enable the 3-D backbone to obtain object boundary features. We use the ONCE dataset to train and validate the proposed model, and the results showed that the proposed CenterPoint-SE achieves 70.33 mAP and an inference speed of 17.15 FPS, outperforming other methods.

Pattern recognition of schizophrenia based on multidimensional spatial feature fusion

Pattern recognition of schizophrenia based on multidimensional spatial feature fusion

Defect detection for industrial neutron radiographic images based on modified YOLO network

Neutron radiography (NR) has been widely used in non-destructive investigations. Since the industrial neutron radiographic images (NRIs) usually suffer from unclear defect features, insufficient data volume, and low efficiency of manual detection, we propose a proof-of-concept defect-detection method based on the modified YOLO network for the degraded NRIs in this paper. Firstly, 6336 radiographic images including crack, inclusion, blow hole, and residual core are built as the defect-detection dataset. Secondly, the types and the relative coordinates of defects are labeled by a graphical image annotation tool (i.e., Labelimg). Finally, the adaptive spatial feature fusion (ASFF) and convolutional block attention module (CBAM) are introduced to the modified YOLO network to enhance its ability of small-size defect detection. Experimental results demonstrate that the proposed method can achieve the average accuracy of 98.1% and detection rate of 85.985 frames per second (FPS) with a single NVIDIA GeForce RTX 4090 GPU on the built radiographic image dataset. In addition, the proposed method shows a good potential in detecting the above defects contained in the real NRIs.

FL-YOLOv7: A Lightweight Small Object Detection Algorithm in Forest Fire Detection

Given the limited computing capabilities of UAV terminal equipment, there is a challenge in balancing the accuracy and computational cost when deploying the target detection model for forest fire detection on the UAV. Additionally, the fire targets photographed by the UAV are small and prone to misdetection and omission during detection. This paper proposes a lightweight, small target detection model, FL-YOLOv7, based on YOLOv7. First, we designed a light module, C3GhostV2, to replace the feature extraction module in YOLOv7. Simultaneously, we used the Ghost module to replace some of the standard convolution layers in the backbone network, accelerating inference speed and reducing model parameters. Secondly, we introduced the Parameter-Free Attention (SimAm) attention mechanism to highlight the features of smoke and fire targets and suppress background interference, improving the model’s representation and generalization performance without increasing network parameters. Finally, we incorporated the Adaptive Spatial Feature Fusion (ASFF) module to address the model’s weak small target detection capability and use the loss function with dynamically adjustable sample weights (WIoU) to weaken the impact of low-quality or complex samples and improve the model’s overall performance. Experimental results show that FL-YOLOv7 reduces the parameter count by 27% compared to the YOLOv7 model while improving 2.9% mAP50small and 24.4 frames per second in FPS, demonstrating the effectiveness and superiority of our model in small target detection, as well as its real-time and reliability in forest fire scenarios.

STFT: Spatial and temporal feature fusion for transformer tracker

AbstractSiamese‐based trackers have demonstrated robust performance in object tracking, while Transformers have achieved widespread success in object detection. Currently, many researchers use a hybrid structure of convolutional neural networks and Transformers to design the backbone network of trackers, aiming to improve performance. However, this approach often underutilises the global feature extraction capability of Transformers. The authors propose a novel Transformer‐based tracker that fuses spatial and temporal features. The tracker consists of a multilayer spatial feature fusion network (MSFFN), a temporal feature fusion network (TFFN), and a prediction head. The MSFFN includes two phases: feature extraction and feature fusion, and both phases are constructed with a Transformer. Compared with the hybrid structure of “CNNs + Transformer,” the proposed method enhances the continuity of feature extraction and the ability of information interaction between features, enabling comprehensive feature extraction. Moreover, to consider the temporal dimension, the authors propose a TFFN for updating the template image. The network utilises the Transformer to fuse the tracking results of multiple frames with the initial frame, allowing the template image to continuously incorporate more information and maintain the accuracy of target features. Extensive experiments show that the tracker STFT achieves state‐of‐the‐art results on multiple benchmarks (OTB100, VOT2018, LaSOT, GOT‐10K, and UAV123). Especially, the tracker STFT achieves remarkable area under the curve score of 0.652 and 0.706 on the LaSOT and OTB100 benchmark respectively.