Local Feature Extraction Research Articles

Correcting Optical Aberration via Depth-Aware Point Spread Functions.

Optical aberration is a ubiquitous degeneration in realistic lens-based imaging systems. Optical aberrations are caused by the differences in the optical path length when light travels through different regions of the camera lens with different incident angles. The blur and chromatic aberrations manifest significant discrepancies when the optical system changes. This work designs a transferable and effective image simulation system of simple lenses via multi-wavelength, depth-aware, spatially-variant four-dimensional point spread functions (4D-PSFs) estimation by changing a small amount of lens-dependent parameters. The image simulation system can alleviate the overhead of dataset collecting and exploiting the principle of computational imaging for effective optical aberration correction. With the guidance of domain knowledge about the image formation model provided by the 4D-PSFs, we establish a multi-scale optical aberration correction network for degraded image reconstruction, which consists of a scene depth estimation branch and an image restoration branch. Specifically, we propose to predict adaptive filters with the depth-aware PSFs and carry out dynamic convolutions, which facilitate the model's generalization in various scenes. We also employ convolution and self-attention mechanisms for global and local feature extraction and realize a spatially-variant restoration. The multi-scale feature extraction complements the features across different scales and provides fine details and contextual features. Extensive experiments demonstrate that our proposed algorithm performs favorably against state-of-the-art restoration methods.

GRDATFusion: A gradient residual dense and attention transformer infrared and visible image fusion network for smart city security systems in cloud and fog computing

AbstractThe infrared and visible fusion technology holds a pivotal position in smart city for cloud and fog computing, particularly in security system. By fusing infrared and visible image information, this technology enhances target identification, tracking and monitoring precision, bolstering overall system security. However, existing deep learning‐based methods rely heavily on convolutional operations, which excel at extracting local features but have limited receptive fields, hampering global information capture. To overcome this difficulty, we introduce GRDATFusion, a novel end‐to‐end network comprising three key modules: transformer, gradient residual dense and attention residual. The gradient residual dense module extracts local complementary features, leveraging a dense‐shaped network to retain potentially lost information. The attention residual module focuses on crucial input image details, while the transformer module captures global information and models long‐range dependencies. Experiments on public datasets show that GRDATFusion outperforms state‐of‐the‐art algorithms in qualitative and quantitative assessments. Ablation studies validate our approach's advantages, and efficiency comparisons demonstrate its computational efficiency. Therefore, our method makes the security systems in smart city with shorter delay and satisfies the real‐time requirement.

Vehicle Classification Algorithm Based on Improved Vision Transformer

Vehicle classification technology is one of the foundations in the field of automatic driving. With the development of deep learning technology, visual transformer structures based on attention mechanisms can represent global information quickly and effectively. However, due to direct image segmentation, local feature details and information will be lost. To solve this problem, we propose an improved vision transformer vehicle classification network (IND-ViT). Specifically, we first design a CNN-In D branch module to extract local features before image segmentation to make up for the loss of detail information in the vision transformer. Then, in order to solve the problem of misdetection caused by the large similarity of some vehicles, we propose a sparse attention module, which can screen out the discernible regions in the image and further improve the detailed feature representation ability of the model. Finally, this paper uses the contrast loss function to further increase the intra-class consistency and inter-class difference of classification features and improve the accuracy of vehicle classification recognition. Experimental results show that the accuracy of the proposed model on the datasets of vehicle classification BIT-Vehicles, CIFAR-10, Oxford Flower-102, and Caltech-101 is higher than that of the original vision transformer model. Respectively, it increased by 1.3%, 1.21%, 7.54%, and 3.60%; at the same time, it also met a certain real-time requirement to achieve a balance of accuracy and real time.

MobileAmcT: A Lightweight Mobile Automatic Modulation Classification Transformer in Drone Communication Systems

With the rapid advancement of wireless communication technology, automatic modulation classification (AMC) plays a crucial role in drone communication systems, ensuring reliable and efficient communication in various non-cooperative environments. Deep learning technology has demonstrated significant advantages in the field of AMC, effectively and accurately extracting and classifying modulation signal features. However, existing deep learning models often have high computational costs, making them difficult to deploy on resource-constrained drone communication devices. To address this issue, this study proposes a lightweight Mobile Automatic Modulation Classification Transformer (MobileAmcT). This model combines the advantages of lightweight convolutional neural networks and efficient Transformer modules, incorporating the Token and Channel Conv (TCC) module and the EfficientShuffleFormer module to enhance the accuracy and efficiency of the automatic modulation classification task. The TCC module, based on the MetaFormer architecture, integrates lightweight convolution and channel attention mechanisms, significantly improving local feature extraction efficiency. Additionally, the proposed EfficientShuffleFormer innovatively improves the traditional Transformer architecture by adopting Efficient Additive Attention and a novel ShuffleConvMLP feedforward network, effectively enhancing the global feature representation and fusion capabilities of the model. Experimental results on the RadioML2016.10a dataset show that compared to MobileNet-V2 (CNN-based) and MobileViT-XS (ViT-based), MobileAmcT reduces the parameter count by 74% and 65%, respectively, and improves classification accuracy by 1.7% and 1.09% under different SNR conditions, achieving an accuracy of 62.93%. This indicates that MobileAmcT can maintain high classification accuracy while significantly reducing the parameter count and computational complexity, clearly outperforming existing state-of-the-art AMC methods and other lightweight deep learning models.

An offset-transformer hierarchical model for point cloud-based resistance spot welding quality classification

Resistance spot welding (RSW) is a widely used welding technology in automotive manufacturing, and weld nugget quality is closely related to the quality of the vehicle body. Offline random checks are largely relied on the quality inspection of weld nuggets, but they have low efficiency and high cost. To address this issue, this paper proposes a deep learning model for RSW weld nugget classification, named the offset-transformer hierarchical model (OFTFHC), which is based on the point cloud data of its appearance shape. OFTFHC uses a hierarchical network structure to gradually expand the receptive field. A local feature module is introduced to extract local features from the point cloud, effectively enabling the recognition of the fine structural features of the resistance spot weld point cloud. A residual ratio module, which is based on MLP_MA and uses max and average functions for feature enhancement, is designed to adapt to the complex spatial structure of the point cloud. The offset-transformer structure is used to learn global context features, thereby enhancing the global feature extraction capability. Through classification experiments on RSW weld nuggets across 5 categories with a total of 1050 samples, OFTFHC achieved an average accuracy of 80.6 %, outperforming existing models. This demonstrates the effectiveness and superiority of the method, making it highly suitable for weld nugget quality control in automotive automation production lines.

LCFE-X: Pluggable Local Channel Feature Extraction model enhancement method for hyperspectral image classification

Spectral information of hyperspectral images offers high-dimensional features for downstream tasks. Local spectral signatures in different wavelength ranges are critical for material identification and pixel classification. In line with conventional vision methods, recent studies have focused on bolstering the spatial modelling capabilities of transformers. However, the focus remains predominantly global when the channel dimension is considered. Current research on enhancing local channel feature extraction is primarily model specific and lacks generalisable methods that can be applied to various models. Therefore, this study introduces a Local Channel Feature Extraction (LCFE) module. It extracts spectral-spatial features and key features within each band group and then integrates the features to obtain local channel features. Subsequently, a pluggable LCFE-X model enhancement method is proposed. This involves performing LCFE at the front end of the model and fusing the extracted features with the original features. Three enhanced hyperspectral image classification models, namely, LCFE-ViT, LCFE-SF, and LCFE-MF, are constructed based on recently proposed models. Extensive experiments demonstrate that the LCFE module effectively extracts local spectral signatures, and the proposed method enhances the performance of the various basic models. The code is available at https://github.com/YiSu1997/LCFE-X.

Automatic crack segmentation model based on multi-branch aggregation transformer

Crack detection plays a crucial role in evaluating the safety and durability of civil infrastructure. However, detecting cracks of uneven intensity in complex backgrounds is challenging. To overcome this problem, we propose a dual decoder network (CSMT) based on a multi-branch aggregation Transformer, which uses residual atrous spatial pyramid pooling (RASPP) and Transformer dual decoding branches to extract local and global features of different structures. To enhance global feature extraction, we designed a multi-branch aggregation Transformer (MAT) that adaptively weights the features of two attention heads from spatial and channel dimensions to achieve intra block feature aggregation between dimensions. Meanwhile, to obtain multi-scale semantic information, we constructed a new decoding branch, RASPP, which embeds a squeeze-and-excitation (SE) module and residual structures into standard ASPP. Finally, we propose a feature adaptive fusion module (FAM) to enhance feature fusion between adjacent layers and codec layers. Many experiments on three benchmark datasets have shown that the proposed CSMT segmentation network provides excellent performance in a variety of complex scenarios.

CroMAM: A Cross-Magnification Attention Feature Fusion Model for Predicting Genetic Status and Survival of Gliomas using Histological Images.

Predicting the gene mutation status in whole slide images (WSI) is crucial for the clinical treatment, cancer management, and research of gliomas. With advancements in CNN and Transformer algorithms, several promising models have been proposed. However, existing studies have paid little attention on fusing multi-magnification information, and the model requires processing all patches from a whole slide image. In this paper, we propose a cross-magnification attention model called CroMAM for predicting the genetic status and survival of gliomas. The CroMAM first utilizes a systematic patch extraction module to sample a subset of representative patches for downstream analysis. Next, the CroMAM applies Swin Transformer to extract local and global features from patches at different magnifications, followed by acquiring high-level features and dependencies among single-magnification patches through the application of a Vision Transformer. Subsequently, the CroMAM exchanges the integrated feature representations of different magnifications and encourage the integrated feature representations to learn the discriminative information from other magnification. Additionally, we design a cross-magnification attention analysis method to examine the effect of cross-magnification attention quantitatively and qualitatively which increases the model's explainability. To validate the performance of the model, we compare the proposed model with other multi-magnification feature fusion models on three tasks in two datasets. Extensive experiments demonstrate that the proposed model achieves state-of-the-art performance in predicting the genetic status and survival of gliomas. The implementation of the CroMAM will be publicly available upon the acceptance of this manuscript at https://github.com/GuoJisen/CroMAM.

Enhancing local representation learning through global–local integration with functional connectivity for EEG-based emotion recognition

Emotion recognition based on electroencephalogram (EEG) signals is crucial in understanding human affective states. Current research has limitations in extracting local features. The representation capabilities of local features are limited, making it difficult to comprehensively capture emotional information. In this study, a novel approach is proposed to enhance local representation learning through global–local integration with functional connectivity for EEG-based emotion recognition. By leveraging the functional connectivity of brain regions, EEG signals are divided into global embeddings that represent comprehensive brain connectivity patterns throughout the entire process and local embeddings that reflect dynamic interactions within specific brain functional networks at particular moments. Firstly, a convolutional feature extraction branch based on the residual network is designed to extract local features from the global embedding. To further improve the representation ability and accuracy of local features, a multidimensional collaborative attention (MCA) module is introduced. Secondly, the local features and patch embedded local embeddings are integrated into the feature coupling module (FCM), which utilizes hierarchical connections and enhanced cross-attention to couple region-level features, thereby enhancing local representation learning. Experimental results on three public datasets show that compared with other methods, this method improves accuracy by 4.92% on the DEAP, by 1.11% on the SEED, and by 7.76% on the SEED-IV, demonstrating its superior performance in emotion recognition tasks.

Gearbox Fault Diagnosis Method in Noisy Environments Based on Deep Residual Shrinkage Networks.

Gearbox fault diagnosis is essential in the maintenance and preventive repair of industrial systems. However, in actual working environments, noise frequently interferes with fault signals, consequently reducing the accuracy of fault diagnosis. To effectively address this issue, this paper incorporates the noise attenuation of the DRSN-CW model. A compound fault detection method for gearboxes, integrated with a cross-attention module, is proposed to enhance fault diagnosis performance in noisy environments. First, frequency domain features are extracted from the public dataset by using the fast Fourier transform (FFT). Furthermore, the cross-attention mechanism model is inserted in the optimal position to improve the extraction and recognition rate of global and local fault features. Finally, noise-related features are filtered through soft thresholds within the network structure to efficiently mitigate noise interference. The experimental results show that, compared to existing network models, the proposed model exhibits superior noise immunity and high-precision fault diagnosis performance.

Fault diagnosis of a wave energy converter gearbox based on an Adam optimized CNN-LSTM algorithm

The complex structure and harsh operating environment of wave energy converters can result in various faults in transmission components. Environmental noise in practical operating situations may obscure the effective information in collected vibration signals, significantly increasing the difficulty of fault diagnosis. This paper presents a fault diagnosis model for the gearbox of the point absorber wave energy converter. The model integrates a convolutional neural network with long short-term memory to realize efficient extraction of local features from signals and enhance the performance in time-series analysis. Moreover, the model incorporates the Adaptive Moment Estimation algorithm to address the situations where gradients within tensors exhibit unstable changes in the model. A rigid-flexible coupled dynamics simulation model is developed to simulate vibration signals used to train and verify the fault diagnosis model. Experimental tests of the proposed model on a vibration dataset acquired from real vibration experiments demonstrate its efficacy in diagnosing various types of faults under interference of operating conditions. Comparative studies with other models demonstrate the superiority of the proposed model in terms of fault feature extraction, learning convergence efficiency, and diagnostic accuracy, indicating that the proposed model can achieve faster and more accurate fault diagnosis of wave energy converter gearboxes.

Identifying rice field weeds from unmanned aerial vehicle remote sensing imagery using deep learning

BackgroundRice field weed object detection can provide key information on weed species and locations for precise spraying, which is of great significance in actual agricultural production. However, facing the complex and changing real farm environments, traditional object detection methods still have difficulties in identifying small-sized, occluded and densely distributed weed instances. To address these problems, this paper proposes a multi-scale feature enhanced DETR network, named RMS-DETR. By adding multi-scale feature extraction branches on top of DETR, this model fully utilizes the information from different semantic feature layers to improve recognition capability for rice field weeds in real-world scenarios.MethodsIntroducing multi-scale feature layers on the basis of the DETR model, we conduct a differentiated design for different semantic feature layers. The high-level semantic feature layer adopts Transformer structure to extract contextual information between barnyard grass and rice plants. The low-level semantic feature layer uses CNN structure to extract local detail features of barnyard grass. Introducing multi-scale feature layers inevitably leads to increased model computation, thus lowering model inference speed. Therefore, we employ a new type of Pconv (Partial convolution) to replace traditional standard convolutions in the model.ResultsCompared to the original DETR model, our proposed RMS-DETR model achieved an average recognition accuracy improvement of 3.6% and 4.4% on our constructed rice field weeds dataset and the DOTA public dataset, respectively. The average recognition accuracies reached 0.792 and 0.851, respectively. The RMS-DETR model size is 40.8 M with inference time of 0.0081 s. Compared with three classical DETR models (Deformable DETR, Anchor DETR and DAB-DETR), the RMS-DETR model respectively improved average precision by 2.1%, 4.9% and 2.4%.DiscussionThis model is capable of accurately identifying rice field weeds in complex real-world scenarios, thus providing key technical support for precision spraying and management of variable-rate spraying systems.

A Hybrid Model of Conformer and LSTM for Ocean Wave Height Prediction

This study proposes a hybrid model (Conformer-LSTM) based on Conformer and Long Short-Term Memory networks (LSTM) to overcome the limitations of existing techniques and enhance the accuracy and generalizability of wave height predictions. The model combines the advantages of self-attention mechanisms and convolutional neural networks. It captures global dependencies through multi-head self-attention and utilizes convolutional layers to extract local features, thereby enhancing the model’s adaptability to dynamic changes in time series. The LSTM component handles long-term dependencies, optimizing the coherence and stability of predictions. Additionally, an adaptive feature fusion weight network is introduced to further improve the model’s recognition and utilization efficiency of key features. Experimental data come from the National Oceanic and Atmospheric Administration buoy data, covering wave height, wind speed, and other data from key maritime areas. Evaluation metrics include Mean Absolute Error (MAE), Root Mean Square Error (RMSE), Mean Absolute Percentage Error (MAPE), and Coefficient of Determination (R2), ensuring a comprehensive assessment of model performance. The results show that the Conformer-LSTM model outperforms traditional LSTM, CNN, and CNN-LSTM models at multiple sites, confirming its potential in wave height prediction.

Enhancing autonomous driving safety: A robust traffic sign detection and recognition model TSD-YOLO

As autonomous driving technology rapidly advances, Traffic Sign Detection and Recognition (TSDR) has become pivotal in ensuring the safety and regulatory compliance of autonomous vehicles. Despite progress, existing technologies struggle under challenging conditions such as adverse weather and complex roadway environments. To overcome these obstacles, we introduce a novel model, TSD-YOLO, which leverages Mamba and YOLO technologies to enhance the accuracy and robustness of traffic sign detection. Our innovative YOLO-MAM dual-branch module merges convolutional layer-based local feature extraction with the long-distance dependency capabilities of the State Space Models (SSMs). We conducted experimental validations using the Tsinghua-Tencent 100K (TT-100K) dataset and the Mapillary Traffic Sign Detection (MTSD) dataset, demonstrating our model’s efficacy across various datasets. Furthermore, cross-dataset validations affirm the model’s exceptional generalization and robustness across diverse environments. This study not only bolsters traffic sign detection and recognition in autonomous driving systems but also paves the way for future advancements in autonomous driving technology.

Feature-preserving adaptive 3D scene multi-model simplification algorithm

In a complex 3D scene, which typically contains multiple models with distinct feature differences, determining how to set appropriate simplification rates for each model so that the 3D scene maintains its characteristics well while achieving the desired simplification rate is a challenging task. To address this, this paper proposes a 3D scene simplification algorithm that integrates neural networks to accomplish this task. The algorithm is divided into two stages: the first stage extracts local and global features of each model in the 3D scene to form feature vectors, and the second stage combines neural networks to construct a multi-layer perceptron that predicts the simplification labels for each model in the 3D scene and calculates the simplification weights for each model based on these labels. Experiments have proven that this algorithm not only maintains a good appearance of the 3D scene but also has a smaller overall simplification error compared to the traditional QEM algorithm.

RNE-DSNet: A Re-parameterization Neighborhood Enhancement-based Dual-Stream Network for CT image recognition

Deep learning models based on Transformer and CNN are current research hotspots. However, there are more time complexity and space complexity in extracting global–local features, the feature extraction ability is insufficient for small lesions in CT images. A Re-parameterization Neighborhood Enhancement-based Dual-Stream Network (RNE-DSNet) is proposed for CT image recognition in this paper. This method improves the ability to recognize small lesions while reducing the Params number and FLOPs, which gives a trade-off between performance and cost. The main works are as following: Firstly, a Re-parameterization Neighborhood Enhancement-based Dual-Stream Network is designed to improve the global–local feature extraction ability. Secondly, the Re-parameterization Group Residual Block is designed in the CNN branch, which improves the local feature extraction ability through multi-scale and multi-branch structures, and the number of the Params and FLOPs are reduced effectively. The Neighborhood Enhancement Transformer Block is designed in the Transformer branch, in which the global attention is calculated by parallel Hadamard product and Dot product, and the global attention maps are enhanced by deep convolution operation to embed local features in global features. Thirdly, the Adaptive-random Mask Cropping method is designed to improve the features extraction ability for small lesions in CT images through heat map and random Mask techniques. RNE-DSNet is compared with other SOTA models. The Accuracy (ACC), Precision (PRE), Recall (REC), F1 score (F1), and Area Under Curve (AUC) values of RNE-DSNet are reached 99.22%, 99.23%, 99.08%, 99.23% and 99.38%, respectively, which verified the effectiveness of the RNE-DSNet model. The number Params and FLOPs are reduced by 62.78% and 84.20%, respectively. The effectiveness of RNE-DSNet is verified by heat map visualization. The performance of RNE-DSNet is generally better than the others methods. There is positive significance for computer aided diagnosis.

Integrated RF-CNN-GRU ensemble for enhanced beef quality classification: A multi-modal approach

Ensuring food safety is crucial for health and economic sustainability. A key indicator of food quality is its smell, which can be assessed using an electronic nose (e-nose). Combining e-nose technology with machine learning methods has shown promise in enhancing classification accuracy. However, traditional methods such as Support Vector Machines (SVM), k-Nearest Neighbors (KNN), and Convolutional Neural Networks (CNN) often fall short in accurately distinguishing between fresh and spoiled beef.To address these limitations, this study proposes a novel hybrid model that integrates Random Forest, CNN for local feature extraction, and Gated Recurrent Unit (GRU) for global feature extraction. Utilizing data from 11 e-nose sensors, this model achieves an unprecedented accuracy of up to 0.9977 in differentiating fresh from spoiled beef across all 12 types of cuts. The hybrid model demonstrates superior performance metrics, including accuracy, precision, recall, F1 score, coefficient of determination (R2), root mean square error (RMSE), and residual predictive deviation (RPD), significantly outperforming traditional approaches in beef quality classification. By incorporating both local and global features, our approach offers innovative insights and marks a significant advancement in machine learning applications for food quality assessment.

Heterogeneous feature learning network for multimodal remote sensing image collaborative classification

ABSTRACT While multimodal remote sensing data processing has garnered growing interests in the geoscience field, how to effectively extract, represent, and fuse heterogeneous features remains challenging. In this article, a novel heterogeneous feature learning network (HFLN) architecture is investigated for multimodal remote sensing collaborative classification, which takes both advantages of long-range feature modelling capability of transformer as well as local feature extraction of convolutional neural network (CNN). First, the global spectral dependencies are extracted from hyperspectral images using the spectral transformer structure. Then, local spatial invariant characteristics are extracted from multimodal remote sensing images by convolutional operation. Next, heterogeneous spectral and spatial characteristics, having significant structural and semantic differences, are dynamically integrated through the feature coupling module in an interactive manner. Finally, the multi-stage network architecture is utilized for extracting hierarchical characteristics, where the numbers of feature maps in CNN and heads in transformer are gradually increased to represent more complex features. Four benchmark remote sensing datasets (i.e. hyperspectral images and multimodal remote sensing datasets) are utilized for experimental analysis, and extensive experimental results have certified the progressiveness of the investigated approach.

Fall detection algorithm based on global and local feature extraction

Falls have become one of the main causes of injury and death among the elderly. A high-accuracy fall detection method can effectively detect falls in the elderly, thereby reducing the probability of injury and mortality. This paper proposes a fall detection algorithm based on global and local feature extraction. Specifically, we design a dual-stream network, with one branch composed of a convolutional neural network and a regional attention module for extracting local features from images. The other branch consists of an improved Transformer for extracting global features from images. The local and global features are then fused using a feature fusion module for classification, enabling fall detection. Experimental results show that the proposed approach achieves accuracies of 99.55% and 99.75% when tested with UP-Fall Detection Dataset and Le2i Fall Detection Dataset.

A High-Precision Identification Method for Maize Leaf Diseases and Pests Based on LFMNet under Complex Backgrounds.

Maize, as one of the most important crops in the world, faces severe challenges from various diseases and pests. The timely and accurate identification of maize leaf diseases and pests is of great significance for ensuring agricultural production. Currently, the identification of maize leaf diseases and pests faces two key challenges: (1) In the actual process of identifying leaf diseases and pests, complex backgrounds can interfere with the identification effect. (2) The subtle features of diseases and pests are difficult to accurately extract. To address these challenges, this study proposes a maize leaf disease and pest identification model called LFMNet. Firstly, the localized multi-scale inverted residual convolutional block (LMSB) is proposed to perform preliminary down-sampling on the image, preserving important feature information for the subsequent extraction of fine disease and pest features in the model structure. Then, the feature localization bottleneck (FLB) is proposed to improve the model's ability to focus on and locate disease and pest characteristics and to reduce interference from complex backgrounds. Subsequently, the multi-hop local-feature fusion architecture (MLFFA) is proposed, which effectively addresses the problem of extracting subtle features by enhancing the extraction and fusion of global and local disease and pest features in images. After training and testing on a dataset containing 19,451 images of maize leaf diseases and pests, the LFMNet model demonstrated excellent performance, with an average identification accuracy of 95.68%, a precision of 95.91%, a recall of 95.78%, and an F1 score of 95.83%. Compared to existing models, it exhibits significant advantages, offering robust technical support for the precise identification of maize diseases and pests.