Improve Generalization Performance Research Articles

Enhancing generalization in genetic programming hyper-heuristics through mini-batch sampling strategies for dynamic workflow scheduling

Genetic Programming Hyper-heuristics (GPHH) have been successfully used to evolve scheduling rules for Dynamic Workflow Scheduling (DWS) as well as other challenging combinatorial optimization problems. The method of sampling training instances has a significant impact on the generalization ability of GPHH, yet they are rarely addressed in existing research. This article aims to fill this gap by proposing a GPHH algorithm with a sampling strategy to thoroughly investigate the impact of six instance sampling strategies on algorithmic generalization, including one rotation strategy, three mini-batch strategies, and two hybrid strategies. Experiments across four scenarios with varying settings reveal that: (1) mini-batch with random sampling can outperform rotation in generalizing to unseen workflow scheduling problems under the same computational cost; (2) employing a hybrid strategy that combines rotation and mini-batch further enhances the generalization ability of GPHH; and (3) mini-batch and hybrid strategies can effectively enable heuristics trained on small-scale training instances generalizing well to large-scale unseen ones. These findings highlight the potential of mini-batch strategies in GPHH, offering improved generalization performance while maintaining diversity and suggesting promising avenues for further exploration in GPHH domains.

Pure large kernel convolutional neural network transformer for medical image registration

Deformable medical image registration is a fundamental and critical task in medical image analysis. Recently, deep learning-based methods have rapidly developed and have shown impressive results in deformable image registration. However, existing approaches still suffer from limitations in registration accuracy or generalization performance. To address these challenges, in this paper, we propose a pure convolutional neural network module (CVTF) to implement hierarchical transformers and enhance the registration performance of medical images. CVTF has a larger convolutional kernel, providing a larger global effective receptive field, which can improve the network’s ability to capture long-range dependencies. In addition, we introduce the spatial interaction attention (SIA) module to compute the interrelationship between the target feature pixel points and all other points in the feature map. This helps to improve the semantic understanding of the model by emphasizing important features and suppressing irrelevant ones. Based on the proposed CVTF and SIA, we construct a novel registration framework named PCTNet. We applied PCTNet to generate displacement fields and register medical images, and we conducted extensive experiments and validation on two public datasets, OASIS and LPBA40. The experimental results demonstrate the effectiveness and generality of our method, showing significant improvements in registration accuracy and generalization performance compared to existing methods. Our code has been available at https://github.com/fz852/PCTNet.

Least Squares Generalization-Memorization Regression

Generalization-Memorization learning endeavors to minimize empirical risk while simultaneously reducing expected risk. Although we often do not pay attention to whether the training samples are accurately memorized during regression, improving generalization performance with better memory is always a goal pursued by regression. To tackle this issue, we introduce two new regression models, Least Squares Generalization-Memorization Regression (LSGMR) and Soft Least Squares Generalization-Memorization Regression (SLSGMR), by introducing the memory kernel learning on Least Squares Support Vector Regression (LSSVR). We conduct tests on these models using synthetic dataset and showcase that the LSSVR model can be viewed as a special case of our proposed model. Our experiments highlight that, for numerous problems, the models incorporating the employed memory mechanisms, LSGMR and SLSGMR, prove highly effective in yielding superior results compared to LSSVR on noise regression.

Improved YOLOv5 Based on Multi-Strategy Integration for Multi-Category Wind Turbine Surface Defect Detection

Wind energy is a renewable resource with abundant reserves, and its sustainable development and utilization are crucial. The components of wind turbines, particularly the blades and various surfaces, require meticulous defect detection and maintenance due to their significance. The operational status of wind turbine generators directly impacts the efficiency and safe operation of wind farms. Traditional surface defect detection methods for wind turbines often involve manual operations, which suffer from issues such as high subjectivity, elevated risks, low accuracy, and inefficiency. The emergence of computer vision technologies based on deep learning has provided a novel approach to surface defect detection in wind turbines. However, existing datasets designed for wind turbine surface defects exhibit overall category scarcity and an imbalance in samples between categories. The algorithms designed face challenges, with low detection rates for small samples. Hence, this study first constructs a benchmark dataset for wind turbine surface defects comprising seven categories that encompass all common surface defects. Simultaneously, a wind turbine surface defect detection algorithm based on improved YOLOv5 is designed. Initially, a multi-scale copy-paste data augmentation method is proposed, introducing scale factors to randomly resize the bounding boxes before copy-pasting. This alleviates sample imbalances and significantly enhances the algorithm’s detection capabilities for targets of different sizes. Subsequently, a dynamic label assignment strategy based on the Hungarian algorithm is introduced that calculates the matching costs by weighing different losses, enhancing the network’s ability to learn positive and negative samples. To address overfitting and misrecognition resulting from strong data augmentation, a two-stage progressive training method is proposed, aiding the model’s natural convergence and improving generalization performance. Furthermore, a multi-scenario negative-sample-guided learning method is introduced that involves incorporating unlabeled background images from various scenarios into training, guiding the model to learn negative samples and reducing misrecognition. Finally, slicing-aided hyper inference is introduced, facilitating large-scale inference for wind turbine surface defects in actual industrial scenarios. The improved algorithm demonstrates a 3.1% increase in the mean average precision (mAP) on the custom dataset, achieving 95.7% accuracy in mAP_50 (the IoU threshold is half of the mAP). Notably, the mAPs for small, medium, and large targets increase by 18.6%, 16.4%, and 6.8%, respectively. The experimental results indicate that the enhanced algorithm exhibits high detection accuracy, providing a new and more efficient solution for the field of wind turbine surface defect detection.

Robust Meta-Representation Learning via Global Label Inference and Classification.

Few-shot learning (FSL) is a central problem in meta-learning, where learners must efficiently learn from few labeled examples. Within FSL, feature pre-training has become a popular strategy to significantly improve generalization performance. However, the contribution of pre-training to generalization performance is often overlooked and understudied, with limited theoretical understanding. Further, pre-training requires a consistent set of global labels shared across training tasks, which may be unavailable in practice. In this work, we address the above issues by first showing the connection between pre-training and meta-learning. We discuss why pre-training yields more robust meta-representation and connect the theoretical analysis to existing works and empirical results. Second, we introduce Meta Label Learning (MeLa), a novel meta-learning algorithm that learns task relations by inferring global labels across tasks. This allows us to exploit pre-training for FSL even when global labels are unavailable or ill-defined. Lastly, we introduce an augmented pre-training procedure that further improves the learned meta-representation. Empirically, MeLa outperforms existing methods across a diverse range of benchmarks, in particular under a more challenging setting where the number of training tasks is limited and labels are task-specific.

Fusion of AI Techniques: A Hybrid Approach for Precise Plant Leaf Disease Classification

Classification of plant leaf diseases is an important step in protecting the world's food supply and agricultural yields. There has been encouraging progress in improving the efficiency and accuracy of plant leaf disease diagnosis via the combination of deep learning methods with artificial intelligence (AI) in recent years. This research introduces a new hybrid strategy, CNN+SVM and CNN+RF, that uses deep learning techniques like Convolutional Neural Networks (CNN) in conjunction with more traditional machine learning algorithms like Random Forest (RF) and Support Vector Machine (SVM). Moreover, two hybrid variants, CNN + RF and CNN + SVM, are proposed to exploit the strengths of both paradigms synergistically. To further improve classification accuracy, the study employs Particle Swarm Optimization (PSO) as a feature selection technique. PSO optimizes the feature subset for each classification model, facilitating the extraction of the most informative features, which leads to better discrimination between healthy and diseased plants. The dataset used for experimentation consists of a comprehensive collection of plant leaf images representing various diseases across multiple plant species. Experimental results demonstrate the efficacy of the proposed hybrid approach compared to individual classification methods. The hybrid models achieve higher accuracy rates and improved generalization performance, showcasing the synergistic benefits of combining AI and deep learning techniques. Furthermore, the feature selection process through PSO contributes significantly to enhancing the classification outcomes, providing insights into the discriminative power of selected features. This research contributes to the advancement of plant leaf disease classification methodologies by offering an innovative hybrid approach that leverages the complementary strengths of AI, deep learning, and feature selection techniques. The study's findings underscore the potential for improving plant leaf disease management strategies, ultimately leading to enhanced crop productivity and sustainable agriculture. The proposed hybrid framework can serve as a blueprint for similar classification tasks in other domains, demonstrating the broader impact of synergizing different AI techniques for improved accuracy and performance. CNN+RF gives 95% accuracy, 93% precision, 96% recall and 94% F1 score, whereas CNN+SVM gives 93% accuracy, 91% precision, 94% recall and 92% F1 score.

Structure-aware neural radiance fields without posed camera

The neural radiance fields (NeRF) for realistic novel view synthesis require camera poses to be pre-acquired by a structure-from-motion (SfM) approach. This two-stage strategy is not convenient to use and degrades the performance because the error in the pose extraction can propagate to the view synthesis. We integrate pose extraction and view synthesis into a jointly optimized process so that they can benefit from each other. For network training, only images are given without pre-known camera poses. The camera poses are obtained by the depth-consistent constraint in which the identical feature in different views has the same world coordinates transformed from the local camera coordinates according to the extracted poses. The depth-consistent constraint is jointly optimized with the pixel color constraint. The poses are represented by a CNN-based deep network, whose input is the related frames. This joint optimization enables NeRF to be aware of the scene’s structure, resulting in improved generalization performance. Experiments on three datasets demonstrate the effectiveness of camera pose estimation and novel view synthesis. Code is available at https://github.com/XTU-PR-LAB/SaNerf.

Reducing trainee mistakes. Better performance with changing to a high-fidelity simulation system?

Postpartum hemorrhage is a significant cause of both maternal morbidity and mortality worldwide and is increasing in incidence. This study aimed to assess improvement and identify shortcomings in trainee performance in different simulation systems in the management of postpartum hemorrhage. To perform a pilot study evaluating and comparing high- and low-fidelity simulation models, assessing improvement in repeated performance with high-fidelity mode and identifying mistakes made assessed using Objective Structured Assessment of Technical Skills and thereby exploring what aspects of emergency management of postpartum hemorrhage should be prioritized in teaching settings and assessing what simulation setup is most effective in achieving competence. This was a prospective randomized, single-blinded, single-institution trial in a population of 17 junior obstetrical trainees at the Charité University Hospital Obstetric Simulation Center in Berlin. Trainees were randomized into 2 groups, with either initial low-fidelity simulation or high-fidelity simulation, followed by repeated assessment of performance, using the high-fidelity model simulation system. Individual simulation sessions were video-recorded and transcribed, and the timing of interventions was documented. Strandardized Objective Structured Assessment of Technical Skills forms were used as a checklist for performance. There was a statistically significant general improvement in performance (P=.02; 24.7-27.2 of 31.0 points; average of 8.7%) in the second cycle of simulation assessment and a statistically significant training effect (P=.043; 24.4-28.4 of 31.0 points; average of 12.9%) in the group that underwent repeat simulation assessment from the initial low-fidelity system to the high-fidelity system compared with the group using the same high-fidelity setup (P=.276; 25.0-25.8 of 31.0; average of 2.4%). There was an improvement in the performance when trainees underwent a repeated cycle of simulation assessment changing from a low-fidelity system to a high-fidelity system. Simulation assessment can identify mistakes and learning gaps that are important for obstetrical trainees. This study found that trainees make the same mistakes, regardless of which simulation model was initially used.

Deepfake detection based on cross-domain local characteristic analysis with multi-domain transformer

Deepfake videos present a significant challenge in the current media landscape. While current deepfake detection methods demonstrate satisfactory performance, there is still room for improvement in their ability to generalize and detect unseen scenarios, particularly those involving imperceptible cues. This paper introduces a novel multi-modal deepfake detection model named SpectraVisionFusion Transformer (SVFT), which incorporates spatial and frequency domain statistical artifacts to improve generalization performance. The SVFT framework uses two different backbone encoder models to take advantage of both spatial and frequency domain cues in video sequences, along with a decoder and classifier, for common cross-attention and classification, respectively. The spatial domain branch uses a convolutional transformer-based encoder to analyze facial visual features. In contrast, the frequency domain branch employs a language transformer encoder. Additionally, we introduce a weighted feature embedding fusion mechanism that integrates spectral-based statistical feature embeddings and visual cues to achieve a more comprehensive and balanced spatial-frequency feature representation. By coordinately analyzing these modalities, our model exhibits improved detection and generalization capabilities in unseen scenarios. Our proposed SVFT model achieved 92.57% and 80.63% accuracy in extensive cross-manipulation and dataset evaluation, respectively, while surpassing the performance of traditional and single-domain-based approaches.

Multi-view universum support vector machines with insensitive pinball loss

Multi-view learning (MVL) is a promising field that seeks to make the most of information shared across different views, ultimately improving generalization performance. Numerous multi-view support vector machines (SVMs) have been proposed and shown great success. However, hinge loss is widely used by existing methods, which makes these SVMs sensitive to noise and unstable when re-sampling. Furthermore, many researchers have demonstrated the effectiveness of universum data for supervised learning. This paper proposes a novel MVL method called multi-view universum support vector machine with insensitive pinball loss (Pin-MvUSVM). Specifically, we first integrate the pinball loss into the multi-view learning framework, which is dependent on the quantization distance and is robust to noise. Then, we utilize universum data to explore prior knowledge for MVL, which improves the generalization performance. Moreover, we develop multi-view universum twin support vector machine with insensitive pinball loss (Pin-MvUTSVM) as a low-complexity alternative to improve computational efficiency. It solves a pair of smaller-scale quadratic programming problems (QPPs) and utilizes the dual form of the model to solve two convex optimization problems for obtaining the prediction classifier. We conduct classification experiments on four multi-view datasets, and our results show that the proposed convex models outperform several state-of-the-art models.

Watching (natural) beauty boosts task performance: testing the nature-as-reward hypothesis.

In two online studies, we tested the "nature-as-reward hypothesis", which suggests that superior cognitive task performance following nature exposure reflects a general performance improvement, driven by the reward value of beautiful things. In both between-subjects experiments, participants viewed either beautiful or less beautiful images for 10s, comprising beautiful mountain photos (vs. less beautiful mountain drawings) in Study 1 and beautiful fractals (vs. less beautiful pixelated images) in Study 2. Following image exposure, participants engaged in a ticking task requiring them to freely tick up to 200 boxes. Participants had to complete four (Study 1) or five (Study 2) of such ticking tasks, with each task being preceded by either a beautiful or less beautiful image. In Study 1, for a subset of participants the ticking task was framed as a game. We found that in Study 1, ticking declined over the ticking rounds when participants had viewed less beautiful line drawings of mountains, while ticking performance remained unchanged over the rounds after seeing beautiful mountain images. However, when the ticking task was framed as a game, there was no significant difference in ticking performance between the two beauty conditions over the four ticking rounds. In Study 2, participants ticked more boxes over all ticking rounds after viewing images of beautiful fractals compared to less beautiful pixelated images. In line with the nature-as-reward hypothesis, these findings show that brief exposures to beautiful (nature) images can motivate to work and that framing tasks as a game can attenuate this beauty advantage.

A Novel Feature Engineering Method Based on Latent Representation Learning for Radiomics: Application in NSCLC Subtype Classification.

Radiomics refers to the high-throughput extraction of quantitative features from medical images, and is widely used to construct machine learning models for the prediction of clinical outcomes, while feature engineering is the most important work in radiomics. However, current feature engineering methods fail to fully and effectively utilize the heterogeneity of features when dealing with different kinds of radiomics features. In this work, latent representation learning is first presented as a novel feature engineering approach to reconstruct a set of latent space features from original shape, intensity and texture features. This proposed method projects features into a subspace called latent space, in which the latent space features are obtained by minimizing a unique hybrid loss function including a clustering-like loss and a reconstruction loss. The former one ensures the separability among each class while the latter one narrows the gap between the original features and latent space features. Experiments were performed on a multi-center non-small cell lung cancer (NSCLC) subtype classification dataset from 8 international open databases. Results showed that compared with four traditional feature engineering methods (baseline, PCA, Lasso and L2,1-norm minimization), latent representation learning could significantly improve the classification performance of various machine learning classifiers on the independent test set (all p<0.001). Further on two additional test sets, latent representation learning also showed a significant improvement in generalization performance. Our research shows that latent representation learning is a more effective feature engineering method, which has the potential to be used as a general technology in a wide range of radiomics researches.

Learning Ordinal-Hierarchical Constraints for Deep Learning Classifiers.

Real-world classification problems may disclose different hierarchical levels where the categories are displayed in an ordinal structure. However, no specific deep learning (DL) models simultaneously learn hierarchical and ordinal constraints while improving generalization performance. To fill this gap, we propose the introduction of two novel ordinal-hierarchical DL methodologies, namely, the hierarchical cumulative link model (HCLM) and hierarchical-ordinal binary decomposition (HOBD), which are able to model the ordinal structure within different hierarchical levels of the labels. In particular, we decompose the hierarchical-ordinal problem into local and global graph paths that may encode an ordinal constraint for each hierarchical level. Thus, we frame this problem as simultaneously minimizing global and local losses. Furthermore, the ordinal constraints are set by two approaches ordinal binary decomposition (OBD) and cumulative link model (CLM) within each global and local function. The effectiveness of the proposed approach is measured on four real-use case datasets concerning industrial, biomedical, computer vision, and financial domains. The extracted results demonstrate a statistically significant improvement to state-of-the-art nominal, ordinal, and hierarchical approaches.

A Novel Learning-Based Trajectory Generation Strategy for a Quadrotor.

In this article, a learning-based trajectory generation framework is proposed for quadrotors, which guarantees real-time, efficient, and practice-reliable navigation by online making human-like decisions via reinforcement learning (RL) and imitation learning (IL). Specifically, inspired by human driving behavior and the perception range of sensors, a real-time local planner is designed by combining learning and optimization techniques, where the smooth and flexible trajectories are online planned efficiently in the observable area. In particular, the key problems in the framework, temporal optimality (time allocation), and spatial optimality (trajectory distribution) are solved by designing an RL policy, which provides human-like commands in real-time (e.g., slower or faster) to achieve better navigation, instead of generating traditional low-level motions. In this manner, real-time trajectories are calculated using convex optimization according to the efficient and accurate decisions of the RL policy. In addition, to improve generalization performance and to accelerate the training, an expert policy and IL are employed in the framework. Compared with existing works, the kernel contribution is to design a real-time practice-oriented intelligent trajectory generation framework for quadrotors, where human-like decision-making and model-based optimization are integrated to plan high-quality trajectories. The results of comparative experiments in known and unknown environments illustrate the superior performance of the proposed trajectory generation strategy in terms of efficiency, smoothness, and flexibility.

Rock-physics-guided machine learning for shear sonic log prediction

The S-wave velocity ([Formula: see text]) is a vital parameter for various petrophysical, geophysical, and geomechanical applications in subsurface characterization. Nevertheless, obtaining shear sonic log is frequently challenging because of its high economic, time, and operating costs. Conventional methods for predicting [Formula: see text] rely on empirical relationships and rock-physics models, which often fall short in accuracy due to their inability to account for the complex factors influencing the relationship between [Formula: see text] and other parameters. We develop a physics-guided machine learning (ML) approach to predict the shear sonic log using various physical parameters (e.g., natural gamma ray, P-wave velocity, density, and resistivity) that can be readily obtained from standard logging suites. Three types of rock-physical constraints combined with three guidance strategies form the various physics-guided models. Specifically, the three constraint models include mudrock line, empirical P- and S-wave velocity relationship, and multiparameter regression from the logging data, and the three guidance strategies involve physics-guided pseudolabels, physics-guided loss function, and transfer learning. To assess the model’s generalization ability and simulate the lack of labeled data in real-world applications, a single well is used as a training well, whereas the remaining four wells are used to blind test in a clastic reservoir. Compared with supervised ML without any constraints, all models incorporating physical constraints demonstrate a significant improvement in prediction accuracy and generalization performance. This underscores the importance of integrating the first-order physical laws into the network training for shear sonic log prediction. The most successful approach combines the multiparameter regression relationship with the physics-guided pseudolabels in this case, resulting in a remarkable 47% reduction in the average root-mean-square error during the blind test.

Multi-task label noise learning for classification

Multi-task classification improves generalization performance via exploiting the correlations between tasks. However, most multi-task learning methods fail to recognize and filter noisy labels for the classification problems with label noises. To address this issue, this paper proposes a novel multi-task label noise learning method based on loss correction, called MTLNL. MTLNL introduces the class-wise denoising (CWD) method for loss decomposition and centroid estimation of the loss function in multi-task learning, and eliminates the impact of label noise by using label flipping rate. It also extends to the multi-task positive-unlabeled (PU) learning domain, which offers better flexibility and generalization performance. Moreover, Nesterov’s method is applied to accelerate the solution of the model. MTLNL is compared with other algorithms on five benchmark datasets, five image datasets, and a multi-task PU dataset to demonstrate its effectiveness.

Optimal margin distribution matrix machine

As a proficient matrix classifier, Support Matrix Machine (SMM) utilizes the structural information within matrices to achieve remarkable classification results. Recent theoretical advancements have emphasized the importance of optimizing margin distribution (MD) for improving generalization performance. Regrettably, this crucial aspect has been neglected within the framework of the SMM. Furthermore, SMM inherits the hinge loss function from Support Vector Machine (SVM), which has a detrimental impact on its robustness to noise interference. To address these issues, we propose a novel matrix classification method called Optimal Margin Distribution Matrix Machine (ODMM) by introducing a novel MD-based loss function. Our model not only leverages the structural information of the matrix through the low-rank condition but also enhances the generalization performance of the classifier by optimizing the margin distribution. Specifically, we derive the MD-based loss function based on the first-order and second-order statistics of the margin, replacing the traditional hinge loss function in SMM. Additionally, to solve the optimization problem of ODMM, we developed an algorithm based on the Alternating Direction Method of Multipliers (ADMM) framework. Finally, we demonstrate the superior performance of our proposed ODMM through comprehensive comparative experimental analysis.

Semi-Supervised Domain Alignment Learning for Single Image Dehazing.

Convolutional neural networks (CNNs) have attracted much research attention and achieved great improvements in single-image dehazing. However, previous learning-based dehazing methods are mainly trained on synthetic data, which greatly degrades their generalization capability on natural hazy images. To address this issue, this article proposes a semi-supervised learning approach for single-image dehazing, where both synthetic and realistic images are leveraged during training. Considering the situation that it is hard to obtain the realistic pairs of hazy and haze-free images, how to utilize the realistic data is not a trivial work. In this article, a domain alignment module is introduced to narrow the distribution distance between synthetic data and realistic hazy images in a latent feature space. Meanwhile, a haze-aware attention module is designed to describe haze densities of different regions in the image, thus adaptively responds for different hazy areas. Furthermore, the dark channel prior is introduced to the framework to improve the quality of the unsupervised learning results by considering the statistical characters of haze-free images. Such a semi-supervised design can significantly address the domain shift issue between the synthetic and realistic data, and improve generalization performance in the real world. Experiments indicate that the proposed method obtains state-of-the-art performance on both public synthetic and realistic hazy images with better visual results.

Application of cluster repeated mini-batch training method to classify electroencephalography for grab and lift tasks.

Modern deep neural network training is based on mini-batch stochastic gradient optimization. While using extensive mini-batches improves the computational parallelism, the small batch training proved that it delivers improved generalization performance and allows a significantly smaller memory, which might also improve machine throughput. However, mini-batch size and characteristics, a key factor for training deep neural networks, has not been sufficiently investigated in training correlated group features and looping with highly complex ones. In addition, the unsupervised learning method clusters the data into different groups with similar properties to make the training process more stable and faster. Then, the supervised learning algorithm was applied with the cluster repeated mini-batch training (CRMT) methods. The CRMT algorithm changed the random minibatch characteristics in the training step into training in order of clusters. Specifically, the self-organizing maps (SOM) were used to cluster the information into n groups based on the dataset's labels Then, neural network models (ANN) were trained with each cluster using the cluster repeated mini-batch training method. Experiments conducted on EEG datasets demonstrate the survey of the proposed method and optimize it. In addition, the results in our research outperform other state-of-the-art methods.

Enhancing generalization performance of anomaly detection-based active sonar classifier

It is critical to automatically discriminate between target and clutter in developing future unmanned underwater surveillance systems. However, due to similar physical experiences between target echoes and clutter, active sonar classification remains a challenging problem. Recently, a novel anomaly detection-based active sonar classifier, bi-sphere anomaly detection (BiSAD), was proposed, demonstrating improved generalization performance over conventional supervised learning-based approaches. In the present study, we provide a brief overview of active sonar datasets and the BiSAD. Then, we explore modifications to BiSAD, aiming at further improving its generalization capabilities. Through examination using in situ active data, we validate the efficacy of these modifications, showing superior generalization performance compared to conventional supervised learning-based classifiers and the original BiSAD.