Feature Learning Research Articles

Empowering Urdu sentiment analysis: an attention-based stacked CNN-Bi-LSTM DNN with multilingual BERT

Sentiment analysis (SA) as a research field has gained popularity among the researcher throughout the globe over the past 10 years. Deep neural networks (DNN) and word vector models are employed nowadays and perform well in sentiment analysis. Among the different deep neural networks utilized for SA globally, Bi-directional long short-term memory (Bi-LSTM), BERT, and CNN models have received much attention. Even though these models can process a wide range of text types, Because DNNs treat different features the same, using these models in the feature learning phase of a DNN model leads to the creation of a feature space with very high dimensionality. We suggest an attention-based, stacked, two-layer CNN-Bi-LSTM DNN to overcome these glitches. After local feature extraction, by applying stacked two-layer Bi-LSTM, our proposed model extracted coming and outgoing sequences by seeing sequential data streams in backward and forward directions. The output of the stacked two-layer Bi-LSTM is supplied to the attention layer to assign various words with varying values. A second Bi-LSTM layer is constructed atop the initial layer in the suggested network to increase performance. Various experiments have been conducted to evaluate the effectiveness of our proposed model on two Urdu sentiment analysis datasets named as UCSA-21 and UCSA, and an accuracies of 83.12% and 78.91% achieved, respectively.

DEST: Difference enhanced-Swin Transformer for remote sensing change detection

ABSTRACT Remote sensing (RS) change detection (CD) has recently achieved remarkable success thanks to convolutional neural networks (CNNs). However, due to the limited receptive fields of CNN models, existing methods are prone to generating pseudo detections or missed detections. In this letter, we propose a difference enhanced-Swin Transformer network (DEST) for accurate and robust change detection in RS images. First, we design a difference enhancement module (DEM) in the feature extraction stage to boost the feature learning of differences for dual-temporal images at each level. Second, to enlarge the receptive fields of networks and capture more changed details, we apply a Swin Transformer module to the difference features to model the global contextual information. Third, to avoid the semantic loss and simultaneously solve the problem of uneven contributions of features at different levels, we design a feature weight fusion module (FWFM) to effectively aggregate multi-level feature difference maps. Extensive experimental results on two publicly available benchmarks demonstrate that the proposed method is superior to some state-of-the-art change detection models.

A Refined and Efficient CNN Algorithm for Remote Sensing Object Detection

Remote sensing object detection (RSOD) plays a crucial role in resource utilization, geological disaster risk assessment and urban planning. Deep learning-based object-detection algorithms have proven effective in remote sensing image studies. However, accurate detection of objects with small size, dense distribution and complex object arrangement remains a significant challenge in the remote sensing field. To address this, a refined and efficient object-detection algorithm (RE-YOLO) has been proposed in this paper for remote sensing images. Initially, a refined and efficient module (REM) was designed to balance computational complexity and feature-extraction capabilities, which serves as a key component of the RE_CSP block. RE_CSP block efficiently extracts multi-scale information, overcoming challenges posed by complex backgrounds. Moreover, the spatial extracted attention module (SEAM) has been proposed in the bottleneck of backbone to promote representative feature learning and enhance the semantic information capture. In addition, a three-branch path aggregation network (TBPAN) has been constructed as the neck network, which facilitates comprehensive fusion of shallow positional information and deep semantic information across different channels, enabling the network with a robust ability to capture contextual information. Extensive experiments conducted on two large-scale remote sensing datasets, DOTA-v1.0 and SCERL, demonstrate that the proposed RE-YOLO outperforms state-of-the-art other object-detection approaches and exhibits a significant improvement in generalization ability.

Geometric neural operators (gnps) for data-driven deep learning in non-euclidean settings

Abstract We introduce Geometric Neural Operators (GNPs) for data-driven deep learning of geometric features for tasks in non-euclidean settings. We present a formulation for accounting for geometric contributions along with practical neural network architectures and factorizations for training. We then demonstrate how GNPs can be used (i) to estimate geometric properties, such as the metric and curvatures of surfaces, (ii) to approximate solutions of geometric partial differential equations on manifolds, and (iii) to solve Bayesian inverse problems for identifying manifold shapes. These results show a few ways GNPs can be used for incorporating the roles of geometry in the data-driven learning of operators.

RlaNet: A Residual Convolution Nested Long-Short-Term Memory Model with an Attention Mechanism for Wind Turbine Fault Diagnosis

This paper proposes a new fault diagnosis model for wind power systems called residual convolution nested long short-term memory network with an attention mechanism (rlaNet). The method first preprocesses the SCADA data through feature engineering, uses the Hermite interpolation method to handle missing data, and uses the mutual information-based dimensionality reduction technique to improve data quality and eliminate redundant information. rlaNet combines residual networks and nested long short-term memory networks to replace traditional convolutional neural networks and standard long short-term memory architectures, thereby improving feature extraction and ensuring the abstractness and depth of the extracted features. In addition, the model emphasizes the weighted learning of spatiotemporal features in the input data, enhances the focus on key features, and improves training efficiency. Experimental results show that rlaNet achieves an accuracy of more than 90% in wind turbine fault diagnosis, showing good robustness. Furthermore, noise simulation experiments verify the model’s resistance to interference, providing a reliable solution for wind turbine fault diagnosis under complex operating conditions.

A Lightweight and Efficient Multimodal Feature Fusion Network for Bearing Fault Diagnosis in Industrial Applications

To address the issues of single-structured feature input channels, insufficient feature learning capabilities in noisy environments, and large model parameter sizes in intelligent diagnostic models for mechanical equipment, a lightweight and efficient multimodal feature fusion convolutional neural network (LEMFN) method is proposed. Compared with existing models, LEMFN captures rich fault features at multiple scales by combining time-domain and frequency-domain signals, thereby enhancing the model’s robustness to noise and improving data adaptability under varying operating conditions. Additionally, the convolutional block attention module (CBAM) and random overlapping sampling technology (ROST) are introduced, and through a feature fusion strategy, the accurate diagnosis of mechanical equipment faults is achieved. Experimental results demonstrate that the proposed method not only possesses high diagnostic accuracy and rapid convergence but also exhibits strong robustness in noisy environments. Finally, a graphical user interface (GUI)-based mechanical equipment fault detection system was developed to promote the practical application of intelligent fault diagnosis in mechanical equipment.

Predicting radiation pneumonitis in lung cancer using machine learning and multimodal features: a systematic review and meta-analysis of diagnostic accuracy.

To evaluate the diagnostic accuracy of machine learning models incorporating multimodal features for predicting radiation pneumonitis in lung cancer through a systematic review and meta-analysis. Relevant studies were identified through a systematic search of PubMed, Web of Science, Embase, and the Cochrane Library from October 2003 to December 2023. Additional studies were located by reviewing bibliographies and relevant websites. Two independent researchers screened titles, abstracts, and full-text articles according to predefined inclusion and exclusion criteria. Data extraction was performed using standardized forms, and study quality was assessed using the Quality Assessment of Diagnostic Accuracy Studies-2 tool. The primary outcomes, including combined sensitivity, specificity, positive likelihood ratio (PLR), negative likelihood ratio (NLR), diagnostic odds ratio (DOR), and area under the curve (AUC), were calculated using STATA MP-64 software(Stata Corporation LLC, College Station, USA) with a random-effects model. Meta-analysis was conducted to synthesize diagnostic accuracy measures, and analyses of heterogeneity and publication bias were performed. A total of 1,406 patients with primary lung cancer were included in this systematic review, drawing data from 9 studies. The pooled analysis revealed a sensitivity of 0.74 [0.58-0.85] and a specificity of 0.91 [0.87-0.95] for machine learning models in diagnosing radiation pneumonitis. The positive likelihood ratio (PLR) was 8.69 [5.21-14.50], the negative likelihood ratio (NLR) was 0.28 [0.16-0.49], and the diagnostic odds ratio (DOR) was 30.73 [11.96-78.97]. The area under the curve (AUC) was 0.93 [0.90-0.95], indicating excellent diagnostic performance. Meta-regression analysis identified that the number of machine learning models, year of publication, and study design contributed to heterogeneity among studies. No evidence of publication bias was found. Overall, machine learning models incorporating multimodal characteristics demonstrated 75% accuracy in predicting moderate to severe radiation pneumonitis. In conclusion, by integrating the current machine learning (ML) algorithm's ability in big data mining, a predictive model can be constructed by combining multi-modal features such as genetics, imaging, and cell factors. By selecting multiple machine learning algorithm frameworks and competing for the best combination model based on research goals, the reliability and accuracy of the radiation pneumonitis prediction model can be greatly improved. PROSPERO (CRD42024497599).

Synergistic use of handcrafted and deep learning features for tomato leaf disease classification.

This research introduces a Computer-Aided Diagnosis-system designed aimed at automated detections & classification of tomato leaf diseases, combining traditional handcrafted features with advanced deep learning techniques. The system's process encompasses preprocessing, feature extraction, feature fusion, and classification. It utilizes enhancement filters and segmentation algorithms to isolate with Regions-of-Interests (ROI) in images tomato leaves. These features based arranged in ABCD rule (Asymmetry, Borders, Colors, and Diameter) are integrated with outputs from a Convolutional Neural Network (CNN) pretrained on ImageNet. To address data imbalance, we introduced a novel evaluation method that has shown to improve classification accuracy by 15% compared to traditional methods, achieving an overall accuracy rate of 92% in field tests. By merging classical feature engineering with modern machine learning techniques under mutual information-based feature fusion, our system sets a new standard for precision in agricultural diagnostics. Specific performance metrics showcasing the effectiveness of our approach in automated detection and classifying of tomato leaf disease.

SAFENet: Semantic-Aware Feature Enhancement Network for unsupervised cross-domain road scene segmentation

Unsupervised cross-domain road scene segmentation has attracted substantial interest because of its capability to perform segmentation on new and unlabeled domains, thereby reducing the dependence on expensive manual annotations. This is achieved by leveraging networks trained on labeled source domains to classify images on unlabeled target domains. Conventional techniques usually use adversarial networks to align inputs from the source and the target in either of their domains. However, these approaches often fall short in effectively integrating information from both domains due to Alignment in each space usually leads to bias problems during feature learning. To overcome these limitations and enhance cross-domain interaction while mitigating overfitting to the source domain, we introduce a novel framework called Semantic-Aware Feature Enhancement Network (SAFENet) for Unsupervised Cross-domain Road Scene Segmentation. SAFENet incorporates the Semantic-Aware Enhancement (SAE) module to amplify the importance of class information in segmentation tasks and uses the semantic space as a new domain to guide the alignment of the source and target domains. Additionally, we integrate Adaptive Instance Normalization with Momentum (AdaIN-M) techniques, which convert the source domain image style to the target domain image style, thereby reducing the adverse effects of source domain overfitting on target domain segmentation performance. Moreover, SAFENet employs a Knowledge Transfer (KT) module to optimize network architecture, enhancing computational efficiency during testing while maintaining the robust inference capabilities developed during training. To further improve the segmentation performance, we further employ Curriculum Learning, a self-training mechanism that uses pseudo-labels derived from the target domain to iteratively refine the network. Comprehensive experiments on three well-known datasets, “Synthia→Cityscapes” and “GTA5→Cityscapes”, demonstrate the superior performance of our method. In-depth examinations and ablation studies verify the efficacy of each module within the proposed method.

Temporal ViT-U-Net Tandem Model: Enhancing Multi-Sensor Land Cover Classification Through Transformer-Based Utilization of Satellite Image Time Series

Abstract. Semantic segmentation is essential in the field of remote sensing because it is used for various applications such as environmental monitoring and land cover classification. Recent advancements aim to collectively classify data from diverse sensors and epochs to improve predictive accuracy. With the availability of vast Satellite Image Time Series (SITS) data, supervised deep learning methods, such as Transformer models, become viable options. This paper introduces the Temporal Vision Transformer(ViT), designed to extract features from SITS. These features, capturing the temporal patterns of land cover classes, are integrated with features derived from aerial imagery to improve land cover classification. Drawing inspiration from the success of transformers in Natural language processing (NLP), Temporal ViT concurrently extracts spatial and temporal information from SITS data using tailored positional encoding strategies. The proposed approach fosters comprehensive feature learning across both domains, facilitating seamless integration of encoded data from SITS into aerial images. Furthermore, a training strategy is proposed that supports the Temporal ViT to focus on classes with a changing appearance over the year. Extensive experiments carried out in this work indicate the enhanced classification performance of Temporal ViT compared to existing state-of-the-art techniques for multi-modal land cover classification. Our model achieves a 3.8% increase in the mean IoU compared to the network solely relying on aerial images.

FireNet: A Lightweight and Efficient Multi-Scenario Fire Object Detector

Fire and smoke detection technologies face challenges in complex and dynamic environments. Traditional detectors are vulnerable to background noise, lighting changes, and similar objects (e.g., clouds, steam, dust), leading to high false alarm rates. Additionally, they struggle with detecting small objects, limiting their effectiveness in early fire warnings and rapid responses. As real-time monitoring demands grow, traditional methods often fall short in smart city and drone applications. To address these issues, we propose FireNet, integrating a simplified Vision Transformer (RepViT) to enhance global feature learning while reducing computational overhead. Dynamic snake convolution (DSConv) captures fine boundary details of flames and smoke, especially in complex curved edges. A lightweight decoupled detection head optimizes classification and localization, ideal for high inter-class similarity and small targets. FireNet outperforms YOLOv8 on the Fire Scene dataset (FSD) with a mAP@0.5 of 80.2%, recall of 78.4%, and precision of 82.6%, with an inference time of 26.7 ms. It also excels on the FSD dataset, addressing current fire detection challenges.

Local–Global Feature Adaptive Fusion Network for Building Crack Detection

Cracks represent one of the most common types of damage in building structures and it is crucial to detect cracks in a timely manner to maintain the safety of the buildings. In general, tiny cracks require focusing on local detail information while complex long cracks and cracks similar to the background require more global features for detection. Therefore, it is necessary for crack detection to effectively integrate local and global information. Focusing on this, a local–global feature adaptive fusion network (LGFAF-Net) is proposed. Specifically, we introduce the VMamba encoder as the global feature extraction branch to capture global long-range dependencies. To enhance the ability of the network to acquire detailed information, the residual network is added as another local feature extraction branch, forming a dual-encoding network to enhance the performance of crack detection. In addition, a multi-feature adaptive fusion (MFAF) module is proposed to integrate local and global features from different branches and facilitate representative feature learning. Furthermore, we propose a building exterior wall crack dataset (BEWC) captured by unmanned aerial vehicles (UAVs) to evaluate the performance of the proposed method used to identify wall cracks. Other widely used public crack datasets are also utilized to verify the generalization of the method. Extensive experiments performed on three crack datasets demonstrate the effectiveness and superiority of the proposed method.

Consistency-driven feature scoring and regularization network for visible–infrared person re-identification

Recently, visible–infrared person re-identification (VI-ReID) has received considerable attention due to its practical importance. A number of methods extract multiple local features to enrich the diversity of feature representations. However, some local features often involve modality-relevant information, leading to deteriorated performance. Moreover, existing methods optimize the models by only considering the samples at each batch while ignoring the learned features at previous iterations. As a result, the features of the same person images drastically change at different training epochs, hindering the training stability. To alleviate the above issues, we propose a novel consistency-driven feature scoring and regularization network (CFSR-Net), which consists of a backbone network, a local feature learning block, a feature scoring block, and a global–local feature fusion block, for VI-ReID. On the one hand, we design a cross-modality consistency loss to highlight modality-irrelevant local features and suppress modality-relevant local features for each modality, facilitating the generation of a reliable compact local feature. On the other hand, we develop a feature consistency regularization strategy (including a momentum class contrastive loss and a momentum distillation loss) to impose consistency regularization on the learning of different levels of features by considering the learned features at historical epochs. This effectively enables smooth feature changes and thus improves the training stability. Extensive experiments on public VI-ReID datasets clearly show the effectiveness of our method against several state-of-the-art VI-ReID methods. Code will be released at https://github.com/cxtjl/CFSR-Net.

A spatiotemporal correlation and attention-based model for pipeline deformation prediction in foundation pit engineering

In foundation pit engineering, the deformation prediction of adjacent pipelines is crucial for construction safety. Existing approaches depend on constitutive models, grey correlation prediction, or traditional feedforward neural networks. Due to the complex hydrological and geological conditions, as well as the nonstationary and nonlinear characteristics of monitoring data, this problem remains a challenge. By formulating the deformation of monitoring points as multivariate time series, a deep learning-based prediction model is proposed, which utilizes the convolutional neural network to extract the spatial dependencies among various monitoring points, and leverages the bi-directional long-short memory unit network to extract temporal features. Notably, an attention mechanism is introduced to adjust the trainable weights of spatial-temporal features extracted in the prediction. The evaluation of a real-world subway project demonstrates that the proposed model has advantages compared with current models, particularly in long-term prediction. It improves the Adjusted R2 index averagely by from 19.4 to 61.6%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\%$$\\end{document} compared with existing models. The proposed model also exhibits a decrease in mean absolute error ranging from 51.5 to 70.3%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\%$$\\end{document} compared to others. Experiments and analyses verify that the spatial-temporal dependencies in time series and the attention learning for spatial-temporal features can improve the prediction of such engineering problems.

DC-YOLO: an improved field plant detection algorithm based on YOLOv7-tiny

Weeding is an important part of agricultural production. With the development of science and technology, automated weeding is regarded as the future development direction, and how to accurately and efficiently detect plants in the field is one of the key points. Corn seedlings and weeds are similar in color, shape and other characteristics, which brings serious challenges to plant detection. In this paper, we propose an improved model based on YOLOv7-tiny, called DC-YOLO. To improve the extraction of key features in the model, we propose Dual Coordinate Attention model (DCA). In addition, we introduce the Content-Aware ReAssembly of FEatures (CARAFE) operator to represent the up-sampling process as a learnable feature reorganization, which enriches the feature information of the sampled images. Finally, we decoupled the detection head to minimize conflicts between features from different tasks. The results show that applying the proposed method to corn and weed datasets, the detection accuracy of the model reaches 95.7% mean Average Precision (mAP@0.5), the computational effort of the model is 13.083 Giga Floating-point Operations (GFLOPs), and the parameter size is 5.223 Millon (M), which is better than the rest of the mainstream light-weight target detection model.

Comprehensive Review of Feature Extraction Techniques for sEMG Signal Classification: From Handcrafted Features to Deep Learning Approaches

Surface Electromyography (sEMG) has become an essential tool in various fields, including prosthetic control and clinical evaluation of the neuromusculoskeletal system. In recent years, the application of machine learning and deep learning techniques to sEMG signal classification has gained significant interest. This survey provides a detailed exploration of feature extraction methods for sEMG classification, from traditional handcrafted features to learned features. ObjectivesThis review aims to provide a comprehensive overview of feature extraction techniques for sEMG signal classification, focusing on both handcrafted and learned features. It seeks to advance research by offering a deeper understanding of fundamental concepts in sEMG signal analysis, along with comparisons and summaries of state-of-the-art approaches. Materials and MethodsThe survey covers various feature extraction techniques used for sEMG classification, including signal acquisition, preprocessing, and the application of conventional machine learning and deep learning classifiers. It offers taxonomies, definitions, and performance comparisons, equipping researchers with a broad understanding of current methodologies. ResultsHandcrafted features combined with traditional machine learning classifiers have demonstrated strong performance, especially with smaller datasets. However, deep learning techniques have shown superior results in many applications, despite challenges related to data availability and model interpretability. The survey highlights key findings regarding the performance of both approaches. ConclusionThis study bridges the gap between traditional and learned feature extraction techniques for sEMG signal classification. It provides a valuable resource for researchers and practitioners, offering insights that can guide future advancements. Key areas for future research include addressing data scarcity in deep learning and improving model interpretability for clinical applications.

Semantics and instance interactive learning for labeling and segmentation of vertebrae in CT images

Automatically labeling and segmenting vertebrae in 3D CT images compose a complex multi-task problem. Current methods progressively conduct vertebra labeling and semantic segmentation, which typically include two separate models and may ignore feature interaction among different tasks. Although instance segmentation approaches with multi-channel prediction have been proposed to alleviate such issues, their utilization of semantic information remains insufficient. Additionally, another challenge for an accurate model is how to effectively distinguish similar adjacent vertebrae and model their sequential attribute. In this paper, we propose a Semantics and Instance Interactive Learning (SIIL) paradigm for synchronous labeling and segmentation of vertebrae in CT images. SIIL models semantic feature learning and instance feature learning, in which the former extracts spinal semantics and the latter distinguishes vertebral instances. Interactive learning involves semantic features to improve the separability of vertebral instances and instance features to help learn position and contour information, during which a Morphological Instance Localization Learning (MILL) module is introduced to align semantic and instance features and facilitate their interaction. Furthermore, an Ordinal Contrastive Prototype Learning (OCPL) module is devised to differentiate adjacent vertebrae with high similarity (via cross-image contrastive learning), and simultaneously model their sequential attribute (via a temporal unit). Extensive experiments on several datasets demonstrate that our method significantly outperforms other approaches in labeling and segmenting vertebrae. Our code is available at https://github.com/YuZhang-SMU/Vertebrae-Labeling-Segmentation

Deep semantics-preserving cross-modal hashing

Cross-modal hashing has been paid widespread attention in recent years due to its outstanding performance in cross-modal data retrieval. Cross-modal hashing can be decomposed into two steps, i.e., the feature learning and the binarization. However, most existing cross-modal hash methods do not take the supervisory information of the data into consideration during binary quantization, and thus often fail to adequately preserve semantic information. To solve these problems, this paper proposes a novel deep cross-modal hashing method called deep semantics-preserving cross-modal hashing (DSCMH), which makes full use of intra and inter-modal semantic information to improve the model's performance. Moreover, by designing a label network for semantic alignment during the binarization process, DSCMH's performance can be further improved. In order to verify the performance of the proposed method, extensive experiments were conducted on four big datasets. The results show that the proposed method is better than most of the existing cross-modal hashing methods. In addition, the ablation experiment shows that the proposed new regularized terms all have positive effects on the model's performances in cross-modal retrieval. The code of this paper can be downloaded from http://www.scholat.com/laizhihui.

Social Learning for the Masses

I consider a plausible role for social learning as implemented in the work of Jasmina Arifovic in a complex macroeconomic environment. The model is DSGE with considerable heterogeneity, enough to approach Gini coefficients for income, wealth, and consumption in the U.S. data. The economy has an ambient stochastic structure, and I consider transition dynamics following exceptionally large shocks like the global financial crisis or the global pandemic. These shocks are large enough to plausibly perturb the economy out of the rational expectations equilibrium associated with more ordinary shocks. How is equilibrium re-established? I argue that a social learning construct may be more appropriate in this environment, as opposed to the econometric learning constructs often used to analyze departures from rational expectations in the literature. I also argue that a “DNA” feature of social learning may have led to relatively fast convergence to rational expectations observed following these large shocks in the U.S. data.

Distinct connectivity patterns between perception and attention-related brain networks characterize dyslexia: machine learning applied to resting-state fMRI

Diagnosis of dyslexia often occurs in late schooling years, leading to academic and psychological challenges. Furthermore, diagnosis is time-consuming, costly, and reliant on arbitrary cutoffs. On the other hand, automated algorithms hold great potential in medical and psychological diagnostics. The aim of the present study was to develop a machine learning tool for the detection of dyslexia in children based on the intrinsic connectivity patterns of different brain networks underlying perception and attention. Here, 117 children (8-12 years old; 58 females; 52 typical readers; TR and 65 children with dyslexia) completed cognitive and reading assessments and underwent 10 minutes of resting-state fMRI. Functional connectivity coefficients between 264 brain regions were used as features for machine learning. Different supervised algorithms were employed for classification of children with and without dyslexia. A classifier trained on dorsal attention network features exhibited the highest performance (accuracy .79, sensitivity .92, specificity .64). Auditory, visual, and fronto-parietal network-based classification showed intermediate accuracy levels (70-75%). These results highlight significant neurobiological differences in brain networks associated with visual attention between TR and children with dyslexia. Distinct neural integration patterns can differentiate dyslexia from typical development, which may be utilized in the future as a biomarker for the presence and/or severity of dyslexia.