Spatial Feature Fusion Research Articles

FAS-Res2net: An Improved Res2net-Based Script Identification Method for Natural Scenes

Problems such as complex image backgrounds, low image quality, diverse text forms, and similar or common character layouts in different script categories in natural scenes pose great challenges to scene script identification. This paper proposes a new Res2Net-based improved script identification method, namely FAS-Res2Net. In the feature extraction part, the feature pyramid network (FPN) module is introduced, which is beneficial to aggregate the geometric feature information extracted by the shallow network and the semantic feature information extracted by the deep network. Integrating the Adaptive Spatial Feature Fusion (ASFF) module is beneficial to obtain local feature information for optimal weight fusion. In addition, the global feature information of the image is extracted by introducing the swin transformer coding block, which makes the extracted feature information more abundant. In the classification part, the convolutional classifier is used to replace the traditional Linear classification, and the classification confidence of each category is output, which improves the identification efficiency. The improved algorithm achieved identification rates of 94.7% and 96.0% on public script identification datasets SIW-13 and CVSI-2015, respectively, which verified the superiority of the method.

GA-YOLO: A Lightweight YOLO Model for Dense and Occluded Grape Target Detection

Picking robots have become an important development direction of smart agriculture, and the position detection of fruit is the key to realizing robot picking. However, the existing detection models have the shortcomings of missing detection and slow detection speed when detecting dense and occluded grape targets. Meanwhile, the parameters of the existing model are too large, which makes it difficult to deploy to the mobile terminal. In this paper, a lightweight GA-YOLO model is proposed. Firstly, a new backbone network SE-CSPGhostnet is designed, which greatly reduces the parameters of the model. Secondly, an adaptively spatial feature fusion mechanism is used to address the issues of difficult detection of dense and occluded grapes. Finally, a new loss function is constructed to improve detection efficiency. In 2022, a detection experiment was carried out on the image data collected in the Bagui rural area of Guangxi Zhuang Autonomous Region, the results demonstrate that the GA-YOLO model has an mAP of 96.87%, detection speed of 55.867 FPS and parameters of 11.003 M. In comparison to the model before improvement, the GA-YOLO model has improved mAP by 3.69% and detection speed by 20.245 FPS. Additionally, the GA-YOLO model has reduced parameters by 82.79%. GA-YOLO model not only improves the detection accuracy of dense and occluded targets but also lessens model parameters and accelerates detection speed.

A Spectral–Spatial Transformer Fusion Method for Hyperspectral Video Tracking

Hyperspectral videos (HSVs) can record more adequate detail clues than other videos, which is especially beneficial in cases of abundant spectral information. Although traditional methods based on correlation filters (CFs) employed to explore spectral information locally achieve promising results, their performances are limited by ignoring global information. In this paper, a joint spectral–spatial information method, named spectral–spatial transformer-based feature fusion tracker (SSTFT), is proposed for hyperspectral video tracking, which is capable of utilizing spectral–spatial features and considering global interactions. Specifically, the feature extraction module employs two parallel branches to extract multiple-level coarse-grained and fine-grained spectral–spatial features, which are fused with adaptive weights. The extracted features are further fused with the context fusion module based on a transformer with the hyperspectral self-attention (HSA) and hyperspectral cross-attention (HCA), which are designed to capture the self-context feature interaction and the cross-context feature interaction, respectively. Furthermore, an adaptive dynamic template updating strategy is used to update the template bounding box based on the prediction score. The extensive experimental results on benchmark hyperspectral video tracking datasets demonstrated that the proposed SSTFT outperforms the state-of-the-art methods in both precision and speed.

Passenger Flow Detection in Subway Stations Based on Improved You Only Look Once Algorithm

Passenger flow detection plays an important role in guaranteeing passenger safety. It contributes to the efficiency of passenger flow control at stations. However, passenger flow detection in subway stations has the problems of target size inconsistency and poor detection effects of small targets. To solve this problem, an improved You Only Look Once algorithm (Improved-YOLO) is proposed based on the YOLOv4 for passenger flow detection in subway stations. The repeatable bidirectional feature fusion (BiFF) module was designed to combine with the adaptively spatial feature fusion (ASFF) module to replace the feature fusion network of the YOLOv4. To verify the effectiveness of the Improved-YOLO, the passenger flow dataset of Nanning Metro Line 1 was used for the experiment. Augmentation and transfer learning were then used to improve the performance of the model. Compared with the YOLOv4, the results showed that the mean average accuracy (mAP) of the Improved-YOLO increased from 89.63% to 92.96%, and the detection time of a single frame image reduced from 61 ms to 52 ms. Compared with other classical models, the Improved-YOLO showed satisfactory performance in passenger flow detection in subway stations. These experimental results can provide reference and theoretical support for real-time passenger flow detection and passenger behavior recognition in subways.

TSBA-YOLO: An Improved Tea Diseases Detection Model Based on Attention Mechanisms and Feature Fusion

Tea diseases have a significant impact on the yield and quality of tea during the growth of tea trees. The shape and scale of tea diseases are variable, and the tea disease targets are usually small, with the intelligent detection processes of tea diseases also easily disturbed by the complex background of the growing region. In addition, some tea diseases are concentrated in the entire area of the leaves, needing to be inferred from global information. Common target detection models are difficult to solve these problems. Therefore, we proposed an improved tea disease detection model called TSBA-YOLO. We use the dataset of tea diseases collected at the Maoshan Tea Factory in China. The self-attention mechanism was used to enhance the ability of the model to obtain global information on tea diseases. The BiFPN feature fusion network and adaptively spatial feature fusion (ASFF) technology were used to improve the multiscale feature fusion of tea diseases and enhance the ability of the model to resist complex background interference. We integrated the Shuffle Attention mechanism to solve the problem of difficult identifications of small-target tea diseases. In addition, we used data-enhancement methods and transfer learning to expand the dataset and relocate the parameters learned from other plant disease datasets to enhance tea diseases detection. Finally, SIoU was used to further improve the accuracy of the regression. The experimental results show that the proposed model is good at solving a series of problems encountered in the intelligent recognition of tea diseases. The detection accuracy is ahead of the mainstream target detection models, and the detection speed reaches the real-time level.

Adaptively spatial feature fusion network: an improved UAV detection method for wheat scab

Adaptively spatial feature fusion network: an improved UAV detection method for wheat scab

Multimodal motor imagery decoding method based on temporal spatial feature alignment and fusion

Objective. A motor imagery-based brain-computer interface (MI-BCI) translates spontaneous movement intention from the brain to outside devices. Multimodal MI-BCI that uses multiple neural signals contains rich common and complementary information and is promising for enhancing the decoding accuracy of MI-BCI. However, the heterogeneity of different modalities makes the multimodal decoding task difficult. How to effectively utilize multimodal information remains to be further studied. Approach. In this study, a multimodal MI decoding neural network was proposed. Spatial feature alignment losses were designed to enhance the feature representations extracted from the heterogeneous data and guide the fusion of features from different modalities. An attention-based modality fusion module was built to align and fuse the features in the temporal dimension. To evaluate the proposed decoding method, a five-class MI electroencephalography (EEG) and functional near infrared spectroscopy (fNIRS) dataset were constructed. Main results and significance. The comparison experimental results showed that the proposed decoding method achieved higher decoding accuracy than the compared methods on both the self-collected dataset and a public dataset. The ablation results verified the effectiveness of each part of the proposed method. Feature distribution visualization results showed that the proposed losses enhance the feature representation of EEG and fNIRS modalities. The proposed method based on EEG and fNIRS modalities has significant potential for improving decoding performance of MI tasks.

Vehicle door frame positioning method for binocular vision robots based on improved YOLOv4

In the task of using robots to capture end-of-life cars, the position of the vehicle door frame needs to be grasped. A fast and accurate positioning of the vehicle door frame is the key to realizing the automatic car grasping process. Traditional methods for locating and grasping scrap cars rely heavily on manual operations and suffer from low grasping efficiency and poor accuracy. Therefore, this paper proposes a binocular vision robot vehicle door frame spatial localization method based on the improved YOLOv4. This method includes a lightweight and efficient feature fusion target detection network in complex environments, and the target detection results are combined with an enhanced SURF feature–matching method to locate the vehicle door frame position. To simplify the network structure, MobileNetv3 is used instead of the backbone network CSPDarknet53, and deep separable convolution is used in the network. To increase the sensitivity of the network to vehicle door frame targets in complex environments, an improved convolutional block attention module is added to the pyramid attention with simple network backbones. Moreover, adaptive spatial feature fusion is introduced into the network to fully use the features at different scales for more effective feature fusion. Compared with YOLOv4, the number of network parameters is reduced by 73.8%, the mAP is improved by 1.35%, and the detection speed is increased by 28.7%. The experimental results demonstrate that the positioning accuracy of the system is 0.745 mm, which meets the positioning measurement error of less than 1 cm required for the vehicle door frame. The paper also compares our findings with other network models. The results show that the method achieves a good balance between detection speed and detection accuracy, satisfying the task of identifying vehicle door frames in complex environments with good detection results.

A Semi-Supervised Method for Real-Time Forest Fire Detection Algorithm Based on Adaptively Spatial Feature Fusion

Forest fires occur frequently around the world, causing serious economic losses and human casualties. Deep learning techniques based on convolutional neural networks (CNN) are widely used in the intelligent detection of forest fires. However, CNN-based forest fire target detection models lack global modeling capabilities and cannot fully extract global and contextual information about forest fire targets. CNNs also pay insufficient attention to forest fires and are vulnerable to the interference of invalid features similar to forest fires, resulting in low accuracy of fire detection. In addition, CNN-based forest fire target detection models require a large number of labeled datasets. Manual annotation is often used to annotate the huge amount of forest fire datasets; however, this takes a lot of time. To address these problems, this paper proposes a forest fire detection model, TCA-YOLO, with YOLOv5 as the basic framework. Firstly, we combine the Transformer encoder with its powerful global modeling capability and self-attention mechanism with CNN as a feature extraction network to enhance the extraction of global information on forest fire targets. Secondly, in order to enhance the model’s focus on forest fire targets, we integrate the Coordinate Attention (CA) mechanism. CA not only acquires inter-channel information but also considers direction-related location information, which helps the model to better locate and identify forest fire targets. Integrated adaptively spatial feature fusion (ASFF) technology allows the model to automatically filter out useless information from other layers and efficiently fuse features to suppress the interference of complex backgrounds in the forest area for detection. Finally, semi-supervised learning is used to save a large amount of manual labeling effort. The experimental results show that the average accuracy of TCA-YOLO improves by 5.3 compared with the unimproved YOLOv5. TCA-YOLO also outperformed in detecting forest fire targets in different scenarios. The ability of TCA-YOLO to extract global information on forest fire targets was much improved. Additionally, it could locate forest fire targets more accurately. TCA-YOLO misses fewer forest fire targets and is less likely to be interfered with by forest fire-like targets. TCA-YOLO is also more focused on forest fire targets and better at small-target forest fire detection. FPS reaches 53.7, which means that the detection speed meets the requirements of real-time forest fire detection.

An Infusion Containers Detection Method Based on YOLOv4 with Enhanced Image Feature Fusion

The detection of infusion containers is highly conducive to reducing the workload of medical staff. However, when applied in complex environments, the current detection solutions cannot satisfy the high demands for clinical requirements. In this paper, we address this problem by proposing a novel method for the detection of infusion containers that is based on the conventional method, You Only Look Once version 4 (YOLOv4). First, the coordinate attention module is added after the backbone to improve the perception of direction and location information by the network. Then, we build the cross stage partial-spatial pyramid pooling (CSP-SPP) module to replace the spatial pyramid pooling (SPP) module, which allows the input information features to be reused. In addition, the adaptively spatial feature fusion (ASFF) module is added after the original feature fusion module, path aggregation network (PANet), to facilitate the fusion of feature maps at different scales for more complete feature information. Finally, EIoU is used as a loss function to solve the anchor frame aspect ratio problem, and this improvement allows for more stable and accurate information of the anchor aspect when calculating losses. The experimental results demonstrate the advantages of our method in terms of recall, timeliness, and mean average precision (mAP).

Breast Tumor Classification in Ultrasound Images by Fusion of Deep Convolutional Neural Network and Shallow LBP Feature.

Breast cancer is one of the most dangerous and common cancers in women which leads to a major research topic in medical science. To assist physicians in pre-screening for breast cancer to reduce unnecessary biopsies, breast ultrasound and computer-aided diagnosis (CAD) have been used to distinguish between benign and malignant tumors. In this study, we proposed a CAD system for tumor diagnosis using a multi-channel fusion method and feature extraction structure based on multi-feature fusion on breast ultrasound (BUS) images. In the pre-processing stage, the multi-channel fusion method completed the color conversion of the BUS image to make it contain richer information. In the feature extraction stage, the pre-trained ResNet50 network was selected as the basic network, and three levels of features were combined based on adaptive spatial feature fusion (ASFF), and finally, the shallow local binary pattern (LBP) texture features were fused. Support vector machine (SVM) was used for comparative analysis. A retrospective analysis was carried out, and 1615 breast tumor images (572 benign and 1043 malignant) confirmed by pathological examinations were collected. After data processing and augmentation, for an independent test set consisting of 874 breast ultrasound images (457 benign and 417 malignant), the accuracy, precision, recall, specificity, F1 score, and AUC of our method were 96.91%, 98.75%, 94.72%, 98.91%, 0.97, and 0.991, respectively. The results show that the integration of shallow LBP texture features and multi-level depth features can more effectively improve the comprehensive performance of breast tumor diagnosis, and has strong clinical application value. Compared with the past methods, our proposed method is expected to realize the automatic diagnosis of breast tumors and provide an auxiliary tool for radiologists to accurately diagnose breast diseases.

Static Gesture Recognition Algorithm Based on Improved YOLOv5s

With the government’s increasing support for the virtual reality (VR)/augmented reality (AR) industry, it has developed rapidly in recent years. Gesture recognition, as an important human-computer interaction method in VR/AR technology, is widely used in the field of virtual reality. The current static gesture recognition technology has several shortcomings, such as low recognition accuracy and low recognition speed. A static gesture recognition algorithm based on improved YOLOv5s is proposed to address these issues. The content-aware re-assembly of features (CARAFE) is used to replace the nearest neighbor up-sampling method in YOLOv5s to make full use of the semantic information in the feature map and improve the recognition accuracy of the model for gesture regions. The adaptive spatial feature fusion (ASFF) method is introduced to filter out useless information and retain useful information for efficient feature fusion. The bottleneck transformer method is initially introduced into the gesture recognition task, reducing the number of model parameters and increasing the accuracy while accelerating the inference speed. The improved algorithm achieved an mAP(mean average precision) of 96.8%, a 3.1% improvement in average accuracy compared with the original YOLOv5s algorithm; the confidence level of the actual detection results was higher than the original algorithm.

Detection of Laodelphax striatellus (small brown planthopper) based on improved YOLOv5

Laodelphax striatellus (small brown planthopper, SBPH) is a pest feeding mainly on rice, wheat and corn. Meanwhile, SBPH transmits plant viruses during feeding, which may take more serious damage to the agriculture. The research of SBPH control has great significance. Most of the traditional SBPH detection and counting problems are conducted manually, which is inefficient. In this paper, we proposed the improved YOLOv5s algorithm to implement the recognition and counting the system of SBPH. We improve the YOLOv5s based on the channel attention and spatial attention, adding the Convolutional Block Attention Module (CBAM) to YOLOv5s. In order to fuse the feature information of different scales to improve the model, we add the Adaptively Spatial Feature Fusion (ASFF). Experiments prove that the improved model volume is not much different from YOLOv5s, which is one-seventh of YOLOv5x, but the mAP of the improved model is approximately 4 % better than YOLOv5s and 2 % better than YOLOv5x. Finally, an APP was developed using Android Studio. We developed image recognition, counting module, real-time recognition and counting module, which were migrated to the mobile phone.

OCT image denoising algorithm based on discrete wavelet transform and spatial domain feature fusion

Optical coherence tomography (OCT) is an emerging optical imaging modality with high resolution and non-invasive, which plays an important role in applications such as material detection and disease diagnosis, especially for ophthalmic retinal diseases such as age-related macular degeneration, diabetic macular edema and choroidal neovascularization. However, since OCT utilizes the coherent interference of light, the generated image is inevitably affected by speckle noise, which blurs the structural information of the image such as layer structure and lesion point, and the low-quality OCT image makes its subsequent application become difficult. To solve this problem, an OCT image denoising fusion based on discrete wavelet transform and spatial domain feature weighting is proposed in this paper. Extensibility experiments show that the proposed algorithm can better remove noise and retain its precise structural information compared with several state-of-the-art OCT image denoising algorithms.

SFAF-MA: Spatial Feature Aggregation and Fusion With Modality Adaptation for RGB-Thermal Semantic Segmentation

The fusion of RGB and thermal images has profound implications for the semantic segmentation of challenging urban scenes, such as those with poor illumination. Nevertheless, existing RGB-T fusion networks pay less attention to modality differences; i.e., RGB and thermal images are commonly fused with fixed weights. In addition, spatial context details are lost during regular extraction operations, inevitably leading to imprecise object segmentation. To improve the segmentation accuracy, a novel network named spatial feature aggregation and fusion with modality adaptation (SFAF-MA) is proposed in this paper. The modality difference adaptive fusion (MDAF) module is introduced to adaptively fuse RGB and thermal images with corresponding weights generated from an attention mechanism. In addition, the spatial semantic fusion (SSF) module is designed to tap into more information by capturing multiscale perceptive fields with dilated convolutions of different rates, and aggregate shallower-level features with rich visual information and deeper-level features with strong semantics. Compared with existing methods on the public MFNet dataset and PST900 dataset, the proposed network significantly improves the segmentation effectiveness. The code is available at https://github.com/hexunjie/SFAF-MA.

STFF-SM: Steganalysis Model Based on Spatial and Temporal Feature Fusion for Speech Streams

The real-time detection of speech steganography in Voice-over-Internet-Protocol (VoIP) scenarios remains an open problem, as it requires steganalysis methods to perform for low-intensity embeddings and short-sample inputs, as well as provide rapid detection results. To address these challenges, this paper presents a novel steganalysis model based on spatial and temporal feature fusion (STFF-SM). Differing from the existing methods, we take both the integer and fractional pitch delays as input, and design subframe-stitch module to organically integrate subframe-wise integer delays and frame-wise fractional pitch delays. Further, we design a spatial fusion module based on pre-activation residual convolution to extract the pitch spatial features and gradually increase their dimensions to discover finer steganographic distortions to enhance the detection effect, where a Group-Squeeze-Weighting block is introduced to alleviate the information loss in the process of increasing the feature dimension. In addition, we design a temporal fusion module to extract pitch temporal features using the stacked LSTM, where a Gated Feed-Forward Network is introduced to learn the interaction between different feature maps while suppressing the features that are not useful for detection. We evaluated the performance of STFF-SM through comprehensive experiments and comparisons with the state-of-the-art solutions. The experimental results demonstrate that STFF-SM can well meet the needs of real-time detection of speech steganography in VoIP streams, and outperforms the existing methods in detection performance, especially with low embedding strengths and short window sizes.

Spatial Pyramid Pooling and Adaptively Feature Fusion based Yolov3 for Traffic Sign Detection

Traffic sign detection is a key part of intelligent assisted driving, but also a challenging task due to the small size and different scales of objects in foreground and closed range. In this paper, we propose a new traffic sign detection scheme: Spatial Pyramid Pooling and Adaptively Spatial Feature Fusion based Yolov3 (SPP and ASFF-Yolov3). In order to integrate the target detail features and environment context features in the feature extraction stage of Yolov3 network, the Spatial Pyramid Pooling module is introduced into the pyramid network of Yolov3. Additionally, Adaptively Spatial Feature Fusion module is added to the target detection phase of the pyramid network of Yolov3 to avoid the interference of different scale features with the process of gradient calculation. Experimental results show the effectiveness of the proposed SPP and ASFF-Yolov3 network, which achieves better detection results than the original Yolov3 network. It can archive real-time inference speed despite inferior to the original Yolov3 network. The proposed scheme will add an option to the solutions of traffic sign detection with real-time inference speed and effective detection results.

R-YOLOv5: A Lightweight Rotational Object Detection Algorithm for Real-Time Detection of Vehicles in Dense Scenes

A lightweight rotational object detection algorithm, R-YOLOv5, is proposed to address the limitations of traditional object detection algorithms that do not consider the diversity of vehicle scales in drone images and fail to obtain information on rotation angles. The proposed algorithm incorporated an angle prediction branch and introduced a circular smooth label (CSL) angle classification method to make it suitable for detection scenarios based on rotational boxes. A cascaded Swin Transformer block (STrB) is used to reduce computational complexity during feature fusion in the backbone network, further enhancing semantic information and global perception capabilities for small objects. A feature enhancement attention module (FEAM) is proposed to improve the utilization of detailed information through local feature self-supervision. An adaptive spatial feature fusion structure (ASFF) is introduced, which employs features extracted from different levels of the backbone network to perform multi-scale feature fusion. The experimental results show that the detection accuracy reaches 84.91% on the Drone-Vehicle dataset and 90.23% on the UCAS-AOD remote sensing dataset. The lightweight model has a parameter count of only 2.02 million and can achieve 82.6 FPS for high-resolution images, which is significantly better than existing lightweight models and more suitable for real-time detection of rotating vehicles in dense scenes, making it suitable for deployment on a large majority of embedded platforms.

An improved UAV target detection algorithm based on ASFF-YOLOv5s.

Object detection in drone-captured scenarios is a recent popular task. Due to the high flight altitude of unmanned aerial vehicle (UAV), the large variation of target scales, and the existence of dense occlusion of targets, in addition to the high requirements for real-time detection. To solve the above problems, we propose a real-time UAV small target detection algorithm based on improved ASFF-YOLOv5s. Based on the original YOLOv5s algorithm, the new shallow feature map is passed into the feature fusion network through multi-scale feature fusion to improve the extraction capability for small target features, and the Adaptively Spatial Feature Fusion (ASFF) is improved to improve the multi-scale information fusion capability. To obtain anchor frames for the VisDrone2021 dataset, we improve the K-means algorithm to obtain four different scales of anchor frames on each prediction layer. The Convolutional Block Attention Module (CBAM) is added in front of the backbone network and each prediction network layer to improve the capture capability of important features and suppress redundant features. Finally, to address the shortcomings of the original GIoU loss function, the SIoU loss function is used to accelerate the convergence of the model and improve accuracy. Extensive experiments conducted on the dataset VisDrone2021 show that the proposed model can detect a wide range of small targets in various challenging environments. At a detection rate of 70.4 FPS, the proposed model obtained a precision value of 32.55%, F1-score of 39.62%, and a mAP value of 38.03%, which improved 2.77, 3.98, and 5.1%, respectively, compared with the original algorithm, for the detection performance of small targets and to meet the task of real-time detection of UAV aerial images. The current work provides an effective method for real-time detection of small targets in UAV aerial photography in complex scenes, and can be extended to detect pedestrians, cars, etc. in urban security surveillance.

Dual-View Spectral and Global Spatial Feature Fusion Network for Hyperspectral Image Classification

For hyperspectral image (HSI) classification, two branch networks generally use the convolution neural networks (CNNs) to extract the spatial features and the long short-term memory (LSTM) to learn the spectral features. However, CNN with a local kernel neglects the global properties of the whole HSI. LSTM doesn’t consider the macroscopic and detailed information of spectra. In this paper, we propose a dual-view spectral and global spatial feature fusion network (DSGSF) to extract the spatial-spectral features for HSI classification, including a spatial subnetwork and a spectral subnetwork. In the spatial subnetwork, we propose a global spatial feature representation model based on the encoder-decoder structure with channel attention and spatial attention to learn the global spatial features. In the spectral subnetwork, we design a dual-view spectral feature aggregation model with view attention to learn the diversity of spectral features. By fusing the two subnetworks, we construct DSGSF to extract the spatial-spectral features of HSI with strong discriminating performance. Experimental results on three public datasets illustrate that the proposed method can achieve competitive results compared with the state-of-the-art methods. Code: https://github.com/RZWang-WH/DSGSF.