Improve Generalization Performance Research Articles

A rolling bearing fault diagnosis method for imbalanced data based on multi-scale self-attention mechanism and novel loss function

Deep learning methods are widely used in the field of rolling bearing fault diagnosis and produce good results when faced with datasets with roughly equal numbers of normal and faulty samples. However, real-world data often has a serious imbalance, with the number of fault samples being significantly less than the number of normal samples. This dataset imbalance challenges the performance of traditional deep learning methods. To address this problem, this paper proposes an efficient imbalanced data rolling bearing fault diagnosis method. The method consists of two parts: a deep learning network based on a multi-scale self-attention mechanism and a novel loss function. In terms of the deep learning network, firstly, the one-dimensional vibration signal is converted into a two-dimensional image through the Gramian angular field. This conversion maximises the inherent feature extraction capability of the network. Subsequently, the multi-scale learning capability of the network is enhanced by implementing different expansion rates for the head of the multi-scale self-attention mechanism. This nuanced approach allows the network to capture the underlying information more efficiently. Finally, the inclusion of Ghost bottlenecks and feature pyramid networks (FPNs) helps to optimise network efficiency and improve generalisation performance. A novel loss function is also proposed to make the method more suitable for imbalanced data. During the training process, the classification of samples in each class is analysed using the recall metric of imbalanced classification and the real-time recall is used as a weight to weaken the dominance of the majority class. This weighting ensures the adaptability of the method to imbalanced datasets. The proposed method is evaluated using rolling bearing datasets from Case Western Reserve University, USA, and Southeast University, China. Comparison results with other state-of-the-art deep learning methods show that the proposed method has a robust performance when dealing with imbalanced data.

Domain Generalization through Latent Distribution Exploration for Motor Imagery EEG Classification

Electroencephalography (EEG)-based Motor Imagery (MI) brain-computer interface (BCI) systems play essential roles in motor function rehabilitation for patients with post-stroke. Existing neural networks for decoding MI EEG face challenges due to nonstationary characteristics and subject-specific variations of EEG data. To address these challenges and improve generalization performance, this study proposes a domain generalization (DG) model that eliminates the need for user-specific calibration in real-life applications. Specifically, the proposed model comprises two branches: the first branch applies several independent decision-making networks to decode and classify subjects’ motor intentions, while the second branch adaptively assigns weights to classification results and fuses them into a comprehensive decision. Both branches utilize EEGNet and ShallowConvNet to extract time-frequency-spatial features. By implementing multiple classification networks, the model can learn a broad range of data distributions from source subjects, which contributes to improved generalization performance on target subjects. The proposed EEG-DG framework was evaluated on BCI Competition IV Dataset 2a, 2b and PhysioNet. Results show that the proposed framework significantly enhances the classification performance of MI EEG, outperforming several state-of-the-art models on all three datasets, underlining its superior efficacy in real-world scenarios and exceptional generalization performance. The source code can be accessed at https://github.com/DrugLover/Multibranch-DG-EEG.

Partial Multi-Label Learning via Exploiting Instance and Label Correlations

The goal of partial multi-label learning is to induce a multi-label classifier from partial multi-label data where each instance is annotated with a number of candidate labels but only a subset of them are valid. Many of the existing studies either fail to fully utilize instance and label correlations to eliminate noisy labels or build an oversimplified multi-label classifier, both of which are unfavorable for the improvement of generalization performance. In this article, we put forward a novel model named Pml-ilc to learn a multi-label classifier from partial multi-label data. Specifically, Pml-ilc first encodes instances and labels into a compact semantic space and takes full advantage of instance and label correlations to eliminate noisy labels. Then, it induces a linear mapping from the feature space to the label space while exploiting label-specific features and instance correlations to facilitate the multi-label classifier learning process. Finally, the above two steps are combined into a joint optimization problem and an efficient alternating optimization procedure is developed to find a satisfactory solution. Extensive experiments show that Pml-ilc achieves superior performance on both real-world and synthetic partial multi-label datasets in terms of different evaluation metrics.

Active learning concerning sampling cost for enhancing AI-enabled building energy system modeling

Machine learning is widely recognized as a promising data-driven modeling technique for the model-based control and optimization of building energy systems. However, the generalizability of data-driven models often faces significant challenges, as the available training data from building operations usually only covers a limited range of working conditions. Active learning can proactively test unseen and informative working conditions to enrich the training set by adding new data samples, leading to improved generalization performance of data-driven models. A novel distance and information density-based sample strategy is developed that accounts for the real-time status of building operation and outdoor environment. Based on Mahalanobis distance, this strategy determines the sampling value of an unlabeled sample (unseen working condition) by assessing its similarity to both the training samples and other unlabeled samples. As collecting sufficiently representative samples can be difficult, costly, and time-consuming, a distance-based sampling cost metric is proposed to compare the efficiency of different sampling methods, considering the detrimental effects of the actively sampling process on the normal operation of building energy systems. This paper presents a comprehensive and in-depth comparison of five active learning methods, including one incorporating the distance-based sampling strategy, by conducting data experiments on the data collected from the cooling towers of a real high-rise building. The results show that active learning can effectively identify informative data samples and improve the generalization performance of data-driven models. The research outcomes are valuable for enhancing AI-enabled data-driven modeling of building energy systems with substantial decreases in costs on data sampling.

A novel stacking ensemble learner for predicting residual strength of corroded pipelines

Accurately assessing the residual strength of corroded oil and gas pipelines is crucial for ensuring their safe and stable operation. Machine learning techniques have shown promise in addressing this challenge due to their ability to handle complex, non-linear relationships in data. Unlike previous studies that primarily focused on enhancing prediction accuracy through the optimization of single models, this work shifts the emphasis to a different approach: stacking ensemble learning. This study applies a stacking model composed of seven base learners and three meta-learners to predict the residual strength of pipelines using a dataset of 453 instances. Automated hyperparameter tuning libraries are utilized to search for optimal hyperparameters. By evaluating various combinations of base learners and meta-learners, the optimal stacking configuration was determined. The results demonstrate that the stacking model, using k-nearest neighbors as the meta-learner alongside seven base learners, delivers the best predictive performance, with a coefficient of determination of 0.959. Compared to individual models, the stacking model also significantly improves generalization performance. However, the stacking model’s effectiveness on low-strength pipelines is limited due to the small sample size. Furthermore, incorporating original features into the second-layer model did not significantly enhance performance, likely because the first-layer model had already extracted most of the critical features. Given the marginal contribution of model optimization to prediction accuracy, this work offers a novel perspective for improving model performance. The findings have important practical implications for the integrity assessment of corroded pipelines.

Improving generalization performance of landslide susceptibility model considering spatial heterogeneity by using the geomorphic label-based LightGBM

Improving generalization performance of landslide susceptibility model considering spatial heterogeneity by using the geomorphic label-based LightGBM

Predicting and optimizing the dimensions of rod in lattice structures fabricated by laser powder bed fusion

Laser powder bed fusion (LPBF) is a highly effective method for manufacturing complex parts and is widely used in industry for creating lattice structures. However, during the manufacturing process, the actual dimensions and shapes of the lattice structure rod often deviate significantly from the design model due to limited understanding of the structure-process-property relationship in LPBF. This makes it difficult to carry out targeted structural design and process planning based on the technical characteristics of LPBF. This paper investigates the relationship between melt pool dimensions and lattice structure rod dimensions through theoretical analyses. Based on melt pool dimensions obtained with the help of numerical simulation, a prediction model for the actual dimensions of lattice structure rod is established. We design experiments to validate the accuracy of the predictive model, using LPBF to fabricate rods with varying inclinations, dimensions, and cross-sectional shapes. The actual dimensions are measured under a microscope and compared with the predicted dimensions. The results indicate a difference of no more than 0.1 mm. Furthermore, the experiments reveal a correlation between scan path curvature and actual dimensions. Based on this correlation, the dimensional prediction model is corrected, resulting in an improved generalization performance of the model for parts with varying scan paths. Based on the dimension prediction model, we propose an optimization method to modify the CAD dimensions of the rod to ensure the consistency between the actual and designed dimensions of the rod.

Leveraging AutoEncoders and chaos theory to improve adversarial example detection

The phenomenon of adversarial examples is one of the most attractive topics in machine learning research these days. These are particular cases that are able to mislead neural networks, with critical consequences. For this reason, different approaches are considered to tackle the problem. On the one side, defense mechanisms, such as AutoEncoder-based methods, are able to learn from the distribution of adversarial perturbations to detect them. On the other side, chaos theory and Lyapunov exponents (LEs) have also been shown to be useful to characterize them. This work proposes the combination of both domains. The proposed method employs these exponents to add more information to the loss function that is used during an AutoEncoder training process. As a result, this method achieves a general improvement in adversarial examples detection performance for a wide variety of attack methods.

Improving Global Generalization and Local Personalization for Federated Learning.

Federated learning aims to facilitate collaborative training among multiple clients with data heterogeneity in a privacy-preserving manner, which either generates the generalized model or develops personalized models. However, existing methods typically struggle to balance both directions, as optimizing one often leads to failure in another. To address the problem, this article presents a method named personalized federated learning via cross silo prototypical calibration (pFedCSPC) to enhance the consistency of knowledge of clients by calibrating features from heterogeneous spaces, which contributes to enhancing the collaboration effectiveness between clients. Specifically, pFedCSPC employs an adaptive aggregation method to offer personalized initial models to each client, enabling rapid adaptation to personalized tasks. Subsequently, pFedCSPC learns class representation patterns on clients by clustering, averages the representations within each cluster to form local prototypes, and aggregates them on the server to generate global prototypes. Meanwhile, pFedCSPC leverages global prototypes as knowledge to guide the learning of local representation, which is beneficial for mitigating the data imbalanced problem and preventing overfitting. Moreover, pFedCSPC has designed a cross-silo prototypical calibration (CSPC) module, which utilizes contrastive learning techniques to map heterogeneous features from different sources into a unified space. This can enhance the generalization ability of the global model. Experiments were conducted on four datasets in terms of performance comparison, ablation study, in-depth analysis, and case study, and the results verified that pFedCSPC achieves improvements in both global generalization and local personalization performance via calibrating cross-source features and strengthening collaboration effectiveness, respectively.

A Novel Learning-Based Trajectory Generation Strategy for a Quadrotor.

In this article, a learning-based trajectory generation framework is proposed for quadrotors, which guarantees real-time, efficient, and practice-reliable navigation by online making human-like decisions via reinforcement learning (RL) and imitation learning (IL). Specifically, inspired by human driving behavior and the perception range of sensors, a real-time local planner is designed by combining learning and optimization techniques, where the smooth and flexible trajectories are online planned efficiently in the observable area. In particular, the key problems in the framework, temporal optimality (time allocation), and spatial optimality (trajectory distribution) are solved by designing an RL policy, which provides human-like commands in real-time (e.g., slower or faster) to achieve better navigation, instead of generating traditional low-level motions. In this manner, real-time trajectories are calculated using convex optimization according to the efficient and accurate decisions of the RL policy. In addition, to improve generalization performance and to accelerate the training, an expert policy and IL are employed in the framework. Compared with existing works, the kernel contribution is to design a real-time practice-oriented intelligent trajectory generation framework for quadrotors, where human-like decision-making and model-based optimization are integrated to plan high-quality trajectories. The results of comparative experiments in known and unknown environments illustrate the superior performance of the proposed trajectory generation strategy in terms of efficiency, smoothness, and flexibility.

Multiple prior representation learning for self-supervised monocular depth estimation via hybrid transformer

Self-supervised monocular depth estimation aims to infer depth information without relying on labeled data. However, the lack of labeled information poses a significant challenge to the model’s representation, limiting its ability to capture the intricate details of the scene accurately. Prior information can potentially mitigate this issue, enhancing the model’s understanding of scene structure and texture. Nevertheless, solely relying on a single type of prior information often falls short when dealing with complex scenes, necessitating improvements in generalization performance. To address these challenges, we introduce a novel self-supervised monocular depth estimation model that leverages multiple priors to bolster representation capabilities across spatial, context, and semantic dimensions. Specifically, we employ a hybrid transformer and a lightweight pose network to obtain long-range spatial priors in the spatial dimension. Then, the context prior attention is designed to improve generalization, particularly in complex structures or untextured areas. In addition, semantic priors are introduced by leveraging semantic boundary loss, and semantic prior attention is supplemented, further refining the semantic features extracted by the decoder. Experiments on three diverse datasets demonstrate the effectiveness of the proposed model. It integrates multiple priors to comprehensively enhance the representation ability, improving the accuracy and reliability of depth estimation. Codes are available at: https://github.com/MVME-HBUT/MPRLNet.

Strategy to expedite gas sensor calibration based on cyclic temperature operation and data augmentation

The gas sensor calibration using data-driven approach requires a large volume of training data. The conventional method demands numerous measurements of response data, consuming substantial manpower and time resources. To expedite and streamline the sensor calibration and model deployment, this study proposes a cost-effective and practical calibration strategy combining data augmentation and data preprocessing. This strategy aims to minimize both the collection time and amount of measurement data necessary while ensuring high identification accuracy and generalization performance. The random cropping technique based on sliding sampling window is developed to expand the sample size by randomly and consecutively resampling new data from the limited raw response data. This approach allows any fixed-length sequence from the raw response curve to serve as the training and test sample, facilitating fast and real-time measurement for online detection and dynamic analysis. Furthermore, data preprocessing techniques using sequence normalization and fast Fourier transform are employed to extract the species and concentration features, thereby mitigating drift-like effects in the response data. The effectiveness of this strategy is demonstrated with real measurement data, showcasing significant improvements in identification accuracy and generalization performance compared to conventional methods.

Enhancing generalization in genetic programming hyper-heuristics through mini-batch sampling strategies for dynamic workflow scheduling

Genetic Programming Hyper-heuristics (GPHH) have been successfully used to evolve scheduling rules for Dynamic Workflow Scheduling (DWS) as well as other challenging combinatorial optimization problems. The method of sampling training instances has a significant impact on the generalization ability of GPHH, yet they are rarely addressed in existing research. This article aims to fill this gap by proposing a GPHH algorithm with a sampling strategy to thoroughly investigate the impact of six instance sampling strategies on algorithmic generalization, including one rotation strategy, three mini-batch strategies, and two hybrid strategies. Experiments across four scenarios with varying settings reveal that: (1) mini-batch with random sampling can outperform rotation in generalizing to unseen workflow scheduling problems under the same computational cost; (2) employing a hybrid strategy that combines rotation and mini-batch further enhances the generalization ability of GPHH; and (3) mini-batch and hybrid strategies can effectively enable heuristics trained on small-scale training instances generalizing well to large-scale unseen ones. These findings highlight the potential of mini-batch strategies in GPHH, offering improved generalization performance while maintaining diversity and suggesting promising avenues for further exploration in GPHH domains.

Pure large kernel convolutional neural network transformer for medical image registration

Deformable medical image registration is a fundamental and critical task in medical image analysis. Recently, deep learning-based methods have rapidly developed and have shown impressive results in deformable image registration. However, existing approaches still suffer from limitations in registration accuracy or generalization performance. To address these challenges, in this paper, we propose a pure convolutional neural network module (CVTF) to implement hierarchical transformers and enhance the registration performance of medical images. CVTF has a larger convolutional kernel, providing a larger global effective receptive field, which can improve the network’s ability to capture long-range dependencies. In addition, we introduce the spatial interaction attention (SIA) module to compute the interrelationship between the target feature pixel points and all other points in the feature map. This helps to improve the semantic understanding of the model by emphasizing important features and suppressing irrelevant ones. Based on the proposed CVTF and SIA, we construct a novel registration framework named PCTNet. We applied PCTNet to generate displacement fields and register medical images, and we conducted extensive experiments and validation on two public datasets, OASIS and LPBA40. The experimental results demonstrate the effectiveness and generality of our method, showing significant improvements in registration accuracy and generalization performance compared to existing methods. Our code has been available at https://github.com/fz852/PCTNet.

Least Squares Generalization-Memorization Regression

Generalization-Memorization learning endeavors to minimize empirical risk while simultaneously reducing expected risk. Although we often do not pay attention to whether the training samples are accurately memorized during regression, improving generalization performance with better memory is always a goal pursued by regression. To tackle this issue, we introduce two new regression models, Least Squares Generalization-Memorization Regression (LSGMR) and Soft Least Squares Generalization-Memorization Regression (SLSGMR), by introducing the memory kernel learning on Least Squares Support Vector Regression (LSSVR). We conduct tests on these models using synthetic dataset and showcase that the LSSVR model can be viewed as a special case of our proposed model. Our experiments highlight that, for numerous problems, the models incorporating the employed memory mechanisms, LSGMR and SLSGMR, prove highly effective in yielding superior results compared to LSSVR on noise regression.

Improved YOLOv5 Based on Multi-Strategy Integration for Multi-Category Wind Turbine Surface Defect Detection

Wind energy is a renewable resource with abundant reserves, and its sustainable development and utilization are crucial. The components of wind turbines, particularly the blades and various surfaces, require meticulous defect detection and maintenance due to their significance. The operational status of wind turbine generators directly impacts the efficiency and safe operation of wind farms. Traditional surface defect detection methods for wind turbines often involve manual operations, which suffer from issues such as high subjectivity, elevated risks, low accuracy, and inefficiency. The emergence of computer vision technologies based on deep learning has provided a novel approach to surface defect detection in wind turbines. However, existing datasets designed for wind turbine surface defects exhibit overall category scarcity and an imbalance in samples between categories. The algorithms designed face challenges, with low detection rates for small samples. Hence, this study first constructs a benchmark dataset for wind turbine surface defects comprising seven categories that encompass all common surface defects. Simultaneously, a wind turbine surface defect detection algorithm based on improved YOLOv5 is designed. Initially, a multi-scale copy-paste data augmentation method is proposed, introducing scale factors to randomly resize the bounding boxes before copy-pasting. This alleviates sample imbalances and significantly enhances the algorithm’s detection capabilities for targets of different sizes. Subsequently, a dynamic label assignment strategy based on the Hungarian algorithm is introduced that calculates the matching costs by weighing different losses, enhancing the network’s ability to learn positive and negative samples. To address overfitting and misrecognition resulting from strong data augmentation, a two-stage progressive training method is proposed, aiding the model’s natural convergence and improving generalization performance. Furthermore, a multi-scenario negative-sample-guided learning method is introduced that involves incorporating unlabeled background images from various scenarios into training, guiding the model to learn negative samples and reducing misrecognition. Finally, slicing-aided hyper inference is introduced, facilitating large-scale inference for wind turbine surface defects in actual industrial scenarios. The improved algorithm demonstrates a 3.1% increase in the mean average precision (mAP) on the custom dataset, achieving 95.7% accuracy in mAP_50 (the IoU threshold is half of the mAP). Notably, the mAPs for small, medium, and large targets increase by 18.6%, 16.4%, and 6.8%, respectively. The experimental results indicate that the enhanced algorithm exhibits high detection accuracy, providing a new and more efficient solution for the field of wind turbine surface defect detection.

Robust Meta-Representation Learning via Global Label Inference and Classification.

Few-shot learning (FSL) is a central problem in meta-learning, where learners must efficiently learn from few labeled examples. Within FSL, feature pre-training has become a popular strategy to significantly improve generalization performance. However, the contribution of pre-training to generalization performance is often overlooked and understudied, with limited theoretical understanding. Further, pre-training requires a consistent set of global labels shared across training tasks, which may be unavailable in practice. In this work, we address the above issues by first showing the connection between pre-training and meta-learning. We discuss why pre-training yields more robust meta-representation and connect the theoretical analysis to existing works and empirical results. Second, we introduce Meta Label Learning (MeLa), a novel meta-learning algorithm that learns task relations by inferring global labels across tasks. This allows us to exploit pre-training for FSL even when global labels are unavailable or ill-defined. Lastly, we introduce an augmented pre-training procedure that further improves the learned meta-representation. Empirically, MeLa outperforms existing methods across a diverse range of benchmarks, in particular under a more challenging setting where the number of training tasks is limited and labels are task-specific.

Fusion of AI Techniques: A Hybrid Approach for Precise Plant Leaf Disease Classification

Classification of plant leaf diseases is an important step in protecting the world's food supply and agricultural yields. There has been encouraging progress in improving the efficiency and accuracy of plant leaf disease diagnosis via the combination of deep learning methods with artificial intelligence (AI) in recent years. This research introduces a new hybrid strategy, CNN+SVM and CNN+RF, that uses deep learning techniques like Convolutional Neural Networks (CNN) in conjunction with more traditional machine learning algorithms like Random Forest (RF) and Support Vector Machine (SVM). Moreover, two hybrid variants, CNN + RF and CNN + SVM, are proposed to exploit the strengths of both paradigms synergistically. To further improve classification accuracy, the study employs Particle Swarm Optimization (PSO) as a feature selection technique. PSO optimizes the feature subset for each classification model, facilitating the extraction of the most informative features, which leads to better discrimination between healthy and diseased plants. The dataset used for experimentation consists of a comprehensive collection of plant leaf images representing various diseases across multiple plant species. Experimental results demonstrate the efficacy of the proposed hybrid approach compared to individual classification methods. The hybrid models achieve higher accuracy rates and improved generalization performance, showcasing the synergistic benefits of combining AI and deep learning techniques. Furthermore, the feature selection process through PSO contributes significantly to enhancing the classification outcomes, providing insights into the discriminative power of selected features. This research contributes to the advancement of plant leaf disease classification methodologies by offering an innovative hybrid approach that leverages the complementary strengths of AI, deep learning, and feature selection techniques. The study's findings underscore the potential for improving plant leaf disease management strategies, ultimately leading to enhanced crop productivity and sustainable agriculture. The proposed hybrid framework can serve as a blueprint for similar classification tasks in other domains, demonstrating the broader impact of synergizing different AI techniques for improved accuracy and performance. CNN+RF gives 95% accuracy, 93% precision, 96% recall and 94% F1 score, whereas CNN+SVM gives 93% accuracy, 91% precision, 94% recall and 92% F1 score.

Structure-aware neural radiance fields without posed camera

The neural radiance fields (NeRF) for realistic novel view synthesis require camera poses to be pre-acquired by a structure-from-motion (SfM) approach. This two-stage strategy is not convenient to use and degrades the performance because the error in the pose extraction can propagate to the view synthesis. We integrate pose extraction and view synthesis into a jointly optimized process so that they can benefit from each other. For network training, only images are given without pre-known camera poses. The camera poses are obtained by the depth-consistent constraint in which the identical feature in different views has the same world coordinates transformed from the local camera coordinates according to the extracted poses. The depth-consistent constraint is jointly optimized with the pixel color constraint. The poses are represented by a CNN-based deep network, whose input is the related frames. This joint optimization enables NeRF to be aware of the scene’s structure, resulting in improved generalization performance. Experiments on three datasets demonstrate the effectiveness of camera pose estimation and novel view synthesis. Code is available at https://github.com/XTU-PR-LAB/SaNerf.

Reducing trainee mistakes. Better performance with changing to a high-fidelity simulation system?

Postpartum hemorrhage is a significant cause of both maternal morbidity and mortality worldwide and is increasing in incidence. This study aimed to assess improvement and identify shortcomings in trainee performance in different simulation systems in the management of postpartum hemorrhage. To perform a pilot study evaluating and comparing high- and low-fidelity simulation models, assessing improvement in repeated performance with high-fidelity mode and identifying mistakes made assessed using Objective Structured Assessment of Technical Skills and thereby exploring what aspects of emergency management of postpartum hemorrhage should be prioritized in teaching settings and assessing what simulation setup is most effective in achieving competence. This was a prospective randomized, single-blinded, single-institution trial in a population of 17 junior obstetrical trainees at the Charité University Hospital Obstetric Simulation Center in Berlin. Trainees were randomized into 2 groups, with either initial low-fidelity simulation or high-fidelity simulation, followed by repeated assessment of performance, using the high-fidelity model simulation system. Individual simulation sessions were video-recorded and transcribed, and the timing of interventions was documented. Strandardized Objective Structured Assessment of Technical Skills forms were used as a checklist for performance. There was a statistically significant general improvement in performance (P=.02; 24.7-27.2 of 31.0 points; average of 8.7%) in the second cycle of simulation assessment and a statistically significant training effect (P=.043; 24.4-28.4 of 31.0 points; average of 12.9%) in the group that underwent repeat simulation assessment from the initial low-fidelity system to the high-fidelity system compared with the group using the same high-fidelity setup (P=.276; 25.0-25.8 of 31.0; average of 2.4%). There was an improvement in the performance when trainees underwent a repeated cycle of simulation assessment changing from a low-fidelity system to a high-fidelity system. Simulation assessment can identify mistakes and learning gaps that are important for obstetrical trainees. This study found that trainees make the same mistakes, regardless of which simulation model was initially used.