Improve Generalization Performance Research Articles

Feature Selection with Weighted Ensemble Ranking for Improved Classification Performance on the CSE-CIC-IDS2018 Dataset

Feature selection is a crucial step in machine learning, aiming to identify the most relevant features in high-dimensional data in order to reduce the computational complexity of model development and improve generalization performance. Ensemble feature-ranking methods combine the results of several feature-selection techniques to identify a subset of the most relevant features for a given task. In many cases, they produce a more comprehensive ranking of features than the individual methods used alone. This paper presents a novel approach to ensemble feature ranking, which uses a weighted average of the individual ranking scores calculated using these individual methods. The optimal weights are determined using a Taguchi-type design of experiments. The proposed methodology significantly improves classification performance on the CSE-CIC-IDS2018 dataset, particularly for attack types where traditional average-based feature-ranking score combinations result in low classification metrics.

Stable Learning via Sparse Variable Independence

The problem of covariate-shift generalization has attracted intensive research attention. Previous stable learning algorithms employ sample reweighting schemes to decorrelate the covariates when there is no explicit domain information about training data. However, with finite samples, it is difficult to achieve the desirable weights that ensure perfect independence to get rid of the unstable variables. Besides, decorrelating within stable variables may bring about high variance of learned models because of the over-reduced effective sample size. A tremendous sample size is required for these algorithms to work. In this paper, with theoretical justification, we propose SVI (Sparse Variable Independence) for the covariate-shift generalization problem. We introduce sparsity constraint to compensate for the imperfectness of sample reweighting under the finite-sample setting in previous methods. Furthermore, we organically combine independence-based sample reweighting and sparsity-based variable selection in an iterative way to avoid decorrelating within stable variables, increasing the effective sample size to alleviate variance inflation. Experiments on both synthetic and real-world datasets demonstrate the improvement of covariate-shift generalization performance brought by SVI.

When Neural Networks Fail to Generalize? A Model Sensitivity Perspective

Domain generalization (DG) aims to train a model to perform well in unseen domains under different distributions. This paper considers a more realistic yet more challenging scenario, namely Single Domain Generalization (Single-DG), where only a single source domain is available for training. To tackle this challenge, we first try to understand when neural networks fail to generalize? We empirically ascertain a property of a model that correlates strongly with its generalization that we coin as "model sensitivity". Based on our analysis, we propose a novel strategy of Spectral Adversarial Data Augmentation (SADA) to generate augmented images targeted at the highly sensitive frequencies. Models trained with these hard-to-learn samples can effectively suppress the sensitivity in the frequency space, which leads to improved generalization performance. Extensive experiments on multiple public datasets demonstrate the superiority of our approach, which surpasses the state-of-the-art single-DG methods by up to 2.55%. The source code is available at https://github.com/DIAL-RPI/Spectral-Adversarial-Data-Augmentation.

Parametric Model for Coaxial Cavity Filter with Combined KCCA and MLSSVR

Aiming at the problems of poor data effectiveness, low modeling accuracy, and weak generalization in the tuning process of microwave cavity filters, a parametric model for coaxial cavity filter using kernel canonical correlation analysis (KCCA) and multioutput least squares support vector regression (MLSSVR) is proposed in this study. First, the low-dimensional tuning data is mapped to the high-dimensional feature space by kernel canonical correlation analysis, and the nonlinear feature vectors are fused by the kernel function; second, the multioutput least squares support vector regression algorithm is used for parametric modeling to solve the problems of low accuracy and poor prediction performance; third, the support vector of the parameter model is optimized by the differential evolution whale algorithm (DWA) to improve the convergence and generalization ability of the model in actual tuning. Finally, the tuning experiments of two cavity filters with different topologies are carried out. The experimental results show that the proposed method has an obvious improvement in generalization performance and prediction accuracy compared with the traditional methods.

Fast sharpness-aware training for periodic time series classification and forecasting

Various deep learning architectures have been developed to capture long-term dependencies in time series data, but challenges such as overfitting and computational time still exist. The recently proposed optimization strategy called Sharpness-Aware Minimization (SAM) optimization prevents overfitting by minimizing a perturbed loss within the nearby parameter space. However, SAM requires doubled training time to calculate two gradients per iteration, hindering its practical application in time series modeling such as real-time assessment. In this study, we demonstrate that sharpness-aware training improves generalization performance by capturing trend and seasonal components of time series data. To avoid the computational burden of SAM, we leverage the periodic characteristics of time series data and propose a new fast sharpness-aware training method called Periodic Sharpness-Aware Time series Training (PSATT) that reuses gradient information from past iterations. Empirically, the proposed method achieves both generalization and time efficiency in time series classification and forecasting without requiring additional computations compared to vanilla optimizers.

Bayesian neural networks with physics‐aware regularization for probabilistic travel time modeling

AbstractThe integration of data‐driven models such as neural networks for high‐consequence decision making has been largely hindered by their lack of predictive power away from training data and their inability to quantify uncertainties often prevalent in engineering applications. This article presents an ensembling method with function‐space regularization, which allows to integrate prior information about the function of interest, thus improving generalization performance, while enabling quantification of aleatory and epistemic uncertainties. This framework is applied to build a probabilistic ambulance travel time predictor, leveraging historical ambulance data provided by the Fire Department of New York City. Results show that the integration of a non‐Gaussian likelihood and prior information from a road network analysis yields appropriate probabilistic predictions of travel times, which could be further leveraged for emergency medical service (EMS) decision making.

Dynamic Grammar Pruning for Program Size Reduction in Symbolic Regression

Grammar is a key input in grammar-based genetic programming. Grammar design not only influences performance, but also program size. However, grammar design and the choice of productions often require expert input as no automatic approach exists. This research work discusses our approach to automatically reduce a bloated grammar. By utilizing a simple Production Ranking mechanism, we identify productions which are less useful and dynamically prune those to channel evolutionary search towards better (smaller) solutions. Our objective in this work was program size reduction without compromising generalization performance. We tested our approach on 13 standard symbolic regression datasets with Grammatical Evolution. Using a grammar embodying a well-defined function set as a baseline, we compare effective genome length and test performance with our approach. Dynamic grammar pruning achieved significantly better genome lengths for all datasets, while significantly improving generalization performance on three datasets, although it worsened in five datasets. When we utilized linear scaling during the production ranking stages (the first 20 generations) the results dramatically improved. Not only were the programs smaller in all datasets, but generalization scores were also significantly better than the baseline in 6 out of 13 datasets, and comparable in the rest. When the baseline was also linearly scaled as well, the program size was still smaller with the Production Ranking approach, while generalization scores dropped in only three datasets without any significant compromise in the rest.

BILSTM-Based Deep Neural Network for Rock-Mass Classification Prediction Using Depth-Sequence MWD Data: A Case Study of a Tunnel in Yunnan, China

Measurement while drilling (MWD) data reflect the drilling rig–rock mass interaction; they are crucial for accurately classifying the rock mass ahead of the tunnel face. Although machine-learning methods can learn the relationship between MWD data and rock mechanics parameters to support rock classification, most current models do not consider the impact of the continuous drilling-sequence process, thereby leading to rock-classification errors, while small and unbalanced field datasets result in poor model performance. We propose a novel deep neural network model based on Bi-directional Long Short-Term Memory (BILSTM) to extract information-related sequences in MWD data and improve the accuracy of the rock-mass classification. Two optimization modules were designed to improve the model’s generalization performance. Stratified K-fold cross-validation was used for model optimization in small and unbalanced datasets. Model validation is based on the MWD dataset of a highway tunnel in Yunnan, China. Multiple metrics show that the prediction ability of the network is significantly better than those of a multilayer perceptron (MLP) and a support-vector machine (SVM), while the model exhibits an improved generalization performance. The accuracy of the network can reach 90%, which is 13% and 15% higher than the MLP and SVM, respectively.

Improving generalization performance of electrocardiogram classification models

Objective. Recently, many electrocardiogram (ECG) classification algorithms using deep learning have been proposed. Because the ECG characteristics vary across datasets owing to variations in factors such as recorded hospitals and the race of participants, the model needs to have a consistently high generalization performance across datasets. In this study, as part of the PhysioNet/Computing in Cardiology Challenge (PhysioNet Challenge) 2021, we present a model to classify cardiac abnormalities from the 12- and the reduced-lead ECGs. Approach. To improve the generalization performance of our earlier proposed model, we adopted a practical suite of techniques, i.e. constant-weighted cross-entropy loss, additional features, mixup augmentation, squeeze/excitation block, and OneCycle learning rate scheduler. We evaluated its generalization performance using the leave-one-dataset-out cross-validation setting. Furthermore, we demonstrate that the knowledge distillation from the 12-lead and large-teacher models improved the performance of the reduced-lead and small-student models. Main results. With the proposed model, our DSAIL SNU team has received Challenge scores of 0.55, 0.58, 0.58, 0.57, and 0.57 (ranked 2nd, 1st, 1st, 2nd, and 2nd of 39 teams) for the 12-, 6-, 4-, 3-, and 2-lead versions of the hidden test set, respectively. Significance. The proposed model achieved a higher generalization performance over six different hidden test datasets than the one we submitted to the PhysioNet Challenge 2020.

Machine Learning to Predict Three Types of Outcomes After Traumatic Brain Injury Using Data at Admission: A Multicenter Study for Development and Validation.

The difficulty of accurately identifying patients who would benefit from promising treatments makes it challenging to prove the efficacy of novel treatments for traumatic brain injury (TBI). Although machine learning is being increasingly applied to this task, existing binary outcome prediction models are insufficient for the effective stratification of TBI patients. The aim of this study was to develop an accurate 3-class outcome prediction model to enable appropriate patient stratification. To this end, retrospective balanced data of 1200 blunt TBI patients admitted to six Japanese hospitals from January 2018 onwards (200 consecutive cases at each institution) were used for model training and validation. We incorporated 21 predictors obtained in the emergency department, including age, sex, six clinical findings, four laboratory parameters, eight CT findings, and an emergency craniotomy. We developed two machine learning models (XGBoost and dense neural network) and logistic regression models to predict 3-class outcomes based on the Glasgow Outcome Scale-Extended (GOSE) at discharge. The prediction models were developed using a training dataset with n = 1000, and their prediction performances were evaluated over two validation rounds on a validation dataset (n = 80) and a test dataset (n = 120) using the bootstrap method. Of the 1200 patients in aggregate, the median patient age was 71 years, 199 (16.7%) exhibited severe TBI, and emergency craniotomy was performed on 104 patients (8.7%). The median length of stay was 13.0 days. The 3-class outcomes were good recovery/moderate disability for 709 patients (59.1%), severe disability/vegetative state in 416 patients (34.7%), and death in 75 patients (6.2%). XGBoost model performed well with 69.5% sensitivity, 82.5% accuracy, and an area under the receiver operating characteristic curve of 0.901 in the final validation. In terms of the receiver operating characteristic curve analysis, the XGBoost outperformed the neural network-based and logistic regression models slightly. In particular, XGBoost outperformed the logistic regression model significantly in predicting severe disability/vegetative state. Although each model predicted favorable outcomes accurately, they tended to miss the mortality prediction. The proposed machine learning model was demonstrated to be capable of accurate prediction of in-hospital outcomes following TBI, even with the three GOSE-based categories. As a result, it is expected to be more impactful in the development of appropriate patient stratification methods in future TBI studies than conventional binary prognostic models. Furthermore, outcomes were predicted based on only clinical data obtained from the emergency department. However, developing a robust model with consistent performance in diverse scenarios remains challenging, and further efforts are needed to improve generalization performance.

Federated Learning with Research Prototypes: Application to Multi-Center MRI-based Detection of Prostate Cancer with Diverse Histopathology.

Early prostate cancer detection and staging from MRI is extremely challenging for both radiologists and deep learning algorithms, but the potential to learn from large and diverse datasets remains a promising avenue to increase their performance within and across institutions. To enable this for prototype-stage algorithms, where the majority of existing research remains, we introduce a flexible federated learning framework for cross-site training, validation, and evaluation of custom deep learning prostate cancer detection algorithms. We introduce an abstraction of prostate cancer groundtruth that represents diverse annotation and histopathology data. We maximize use of this groundtruth if and when they are available using UCNet, a custom 3D UNet that enables simultaneous supervision of pixel-wise, region-wise, and gland-wise classification. We leverage these modules to perform cross-site federated training using 1400+ heterogeneous multi-parameteric prostate MRI exams from two University hospitals. We observe a positive result, with significant improvements in cross-site generalization performance with negligible intra-site performance degradation for both lesion segmentation and per-lesion binary classification of clinically-significant prostate cancer. Cross-site lesion segmentation performance intersection-over-union (IoU) improved by 100%, while cross-site lesion classification performance overall accuracy improved by 9.5-14.8%, depending on the optimal checkpoint selected by each site. Federated learning can improve the generalization performance of prostate cancer detection models across institutions while protecting patient health information and institution-specific code and data. However, even more data and participating institutions are likely required to improve the absolute performance of prostate cancer classification models. To enable adoption of federated learning with limited re-engineering of federated components, we open-source our FLtools system at https://federated.ucsf.edu, including examples that can be easily adapted to other medical imaging deep learning projects.

Multitask learning for acoustic scene classification with topic-based soft labels and a mutual attention mechanism

Acoustic scene classification (ASC) is a fundamental task of computational sound scene analysis that aims to identify the acoustic environment via audio. Many multitask learning (MTL) models have been proposed in computational sound scene analysis, but most of them are for acoustic event detection (AED). Existing MTL models for ASC usually leverage the knowledge of the primary and auxiliary tasks only via the shared layers and train the network using hard labels. They do not take advantage of the information contained in the primary and auxiliary tasks to improve the generalization performance, and ignore modeling the relationship between events, scenes or groups. Moreover, some models have the problem of subjectivity since they generate labels via observations, and subjectivity can create unreasonable information, which may restrict the improvement of system performance. To address these issues, we propose a novel MTL scheme for ASC that employs a mutual attention mechanism to explore the information contained in the primary and auxiliary tasks and employs a neural topic model to generate soft group labels automatically. The proposed method can model the relationship between groups and allows the primary and auxiliary tasks to make full use of each other’s information to improve generalization performance. Experimental results on two real-world datasets show that our MTL scheme can make full use of the auxiliary task to improve the performance of the ASC primary task and achieves significant improvements compared to baselines.

Easy Ensemble: Simple Deep Ensemble Learning for Sensor-Based Human Activity Recognition

Sensor-based human activity recognition (HAR) is a paramount technology in the Internet of Things services. HAR using representation learning, which automatically learns a feature representation from raw data, is the mainstream method because it is difficult to interpret relevant information from raw sensor data to design meaningful features. Ensemble learning is a robust approach to improve generalization performance; however, deep ensemble learning requires various procedures, such as data partitioning and training multiple models, which are time-consuming and computationally expensive. In this study, we propose an easy ensemble (EE) for HAR, which enables the easy implementation of deep ensemble learning in a single model. In addition, we propose various techniques (input variationer, stepwise ensemble, and channel shuffle) for the EE. Experiments on a benchmark data set for HAR demonstrated the effectiveness of EE and various techniques and their characteristics compared with conventional ensemble learning methods.

Enhanced task parameterized dynamic movement primitives by GMM to solve manipulation tasks

PurposeUnder the development trend of intelligent manufacturing, the unstructured environment requires the robot to have a good generalization performance to adapt to the scene changes. The purpose of this paper aims to present a learning from demonstration (LfD) method (task parameterized [TP]-dynamic movement primitives [DMP]-GMR) that combines DMPs and TP-LfD to improve generalization performance and solve object manipulation tasks.Design/methodology/approachThe dynamic time warping algorithm is applied to processing demonstration data to obtain a more standard learning model in the proposed method. The DMPs are used to model the basic trajectory learning model. The Gaussian mixture model is introduced to learn the force term of DMPs and solve the problem of learning from multiple demonstration trajectories. The robot can learn more local geometric features and generalize the learned model to unknown situations by adding task parameters.FindingsAn evaluation criterion based on curve similarity calculated by the Frechet distance was constructed to evaluate the model’s interpolation and extrapolation performance. The model’s generalization performance was assessed on 2D virtual data sets, and first, the results show that the proposed method has better interpolation and extrapolation performance than other methods.Originality/valueThe proposed model was applied to the axle-hole assembly task on real robots, and the robot’s posture in grasping and placing the axle part was taken as the task parameter of the model. The experiment results show that The proposed model is competitive with other models.

Estimator for generalization performance of machine learning model trained by biased data collected from multiple references

AbstractThe data acquired in civil engineering tasks often involve high acquisition costs, and the available datasets tend to have a limited number of samples and are highly biased. To estimate the performance of machine learning models, k‐fold cross‐validation (k‐CV) is widely used. However, if only limited data are available and the data distribution is biased, k‐CV tends to overestimate the performance for practical applications. This study proposed a new estimator, leave one reference out and k‐CV (LORO‐k‐CV), to determine the practical performance of machine learning models, that is, the generalization performance for population data in the target task, in case data are collected by multiple references resulting in biased data. LORO‐k‐CV is a combination of a new concept, LORO‐CV, that estimates the performance in the extrapolation region of the training data without human intervention and k‐CV, considering the ratio of the interpolation and extrapolation regions. The efficacy of LORO‐k‐CV was validated with its application to the regression task for the chloride‐ion concentration of concrete structures. To more specifically demonstrate the advantages of LORO‐k‐CV in model construction, the feature selections were conducted using both k‐CV and LORO‐k‐CV methods. These results revealed that LORO‐k‐CV can effectively construct a model with improved generalization performance even from the same data in cases where data are collected by multiple references, resulting in biased data.

Semi-supervised atmospheric component learning in low-light image problem.

Ambient lighting conditions play a crucial role in determining the perceptual quality of images from photographic devices. In general, inadequate transmission light and undesired atmospheric conditions jointly degrade the image quality. If we know the desired ambient factors associated with the given low-light image, we can recover the enhanced image easily. Typical deep networks perform enhancement mappings without investigating the light distribution and color formulation properties. This leads to a lack of image instance-adaptive performance in practice. On the other hand, physical model-driven schemes suffer from the need for inherent decompositions and multiple objective minimizations. Moreover, the above approaches are rarely data efficient or free of postprediction tuning. Influenced by the above issues, this study presents a semisupervised training method using no-reference image quality metrics for low-light image restoration. We incorporate the classical haze distribution model to explore the physical properties of the given image to learn the effect of atmospheric components and minimize a single objective for restoration. We validate the performance of our network for six widely used low-light datasets. Experimental studies show that our proposed study achieves a competitive performance for no-reference metrics compared to current state-of-the-art methods. We also show the improved generalization performance of our proposed method which is efficient in preserving face identities in extreme low-light scenarios.

К ПОВЫШЕНИЮ НАДЕЖНОСТИ СЕЛЬСКОХОЗЯЙСТВЕННЫХ ПРИЦЕПОВ

A method for improving the reliability of agricultural trailers by replacing steel sides with composite sandwich structures is considered, which makes it possible to reduce their weight with a general improvement in performance. The fastening of such boards to the metal frame of the platform is carried out using steel supports. Carrying out loading and unloading operations can be carried out by overturning the side board around the lower hinged supports, with the opening of the latches on the two upper supports. A comparative analysis of the existing methods of joining "steel-composite" (adhesive, with the help of mechanical fasteners, combined, glue-pin) made it possible to identify their shortcomings when using a multilayer composite structure characterized by insufficient strength of the inner layers adjacent to the support. The design of the lower hinged support, devoid of the described shortcomings, has been developed. The connection is carried out using two steel lugs containing profiles with spearshaped fasteners, which are introduced into the skin prepregs at the stage of composite manufacturing. The lugs are tightened with bolts, which allows you to create a reliable connection between the steel hinged supports and the side plate, which consists of a multilayer composite. A method for calculating the lower support has been developed, as a result of which the most loaded fasteners have been identified, and an expression has been obtained to determine the maximum acting force. Recommendations are proposed for calculating the strength of the most loaded fastener according to the allowable shear stresses of its cylindrical part.

An improved AdaBoost algorithm for identification of lung cancer based on electronic nose

The research developed an improved intelligent enhancement learning algorithm based on AdaBoost, that can be applied for lung cancer breath detection by the electronic nose (eNose). First, collected the breath signals from volunteers by eNose, including healthy individuals and people who had lung cancer. Additionally, the signals' features were extracted and optimized. Then, multi sub-classifiers were obtained, and their coefficients were derived from the training error. To improve generalization performance, K-fold cross-validation was used when constructing each sub-classifier. The prediction results of a sub-classifier on the test set were then achieved by the voting method. Thus, an improved AdaBoost classifier would be built through heterogeneous integration. The results shows that the average precision of the improved algorithm classifier for distinguishing between people with lung cancer and healthy individuals could reach 98.47%, with 98.33% sensitivity and 97% specificity. And in 100 independent and randomized tests, the coefficient of variation of the classifier's performance hardly exceeded 4%. Compared with other integrated algorithms, the generalization and stability of the improved algorithm classifier are more superior. It is clear that the improved AdaBoost algorithm may help screen out lung cancer more comprehensively. Additionally, it will significantly advance the use of eNose in the early identification of lung cancer.

Quantum Mechanics/Molecular Mechanics Simulations on NVIDIA and AMD Graphics Processing Units.

We have ported and optimized the graphics processing unit (GPU)-accelerated QUICK and AMBER-based ab initio quantum mechanics/molecular mechanics (QM/MM) implementation on AMD GPUs. This encompasses the entire Fock matrix build and force calculation in QUICK including one-electron integrals, two-electron repulsion integrals, exchange-correlation quadrature, and linear algebra operations. General performance improvements to the QUICK GPU code are also presented. Benchmarks carried out on NVIDIA V100 and AMD MI100 cards display similar performance on both hardware for standalone HF/DFT calculations with QUICK and QM/MM molecular dynamics simulations with QUICK/AMBER. Furthermore, with respect to the QUICK/AMBER release version 21, significant speedups are observed for QM/MM molecular dynamics simulations. This significantly increases the range of scientific problems that can be addressed with open-source QM/MM software on state-of-the-art computer hardware.

A Semi-Supervised Approach for Improving Generalization in Non-Intrusive Load Monitoring.

Non-intrusive load monitoring (NILM) considers different approaches for disaggregating energy consumption in residential, tertiary, and industrial buildings to enable smart grid services. The main feature of NILM is that it can break down the bulk electricity demand, as recorded by conventional smart meters, into the consumption of individual appliances without the need for additional meters or sensors. Furthermore, NILM can identify when an appliance is in use and estimate its real-time consumption based on its unique consumption patterns. However, NILM is based on machine learning methods and its performance is dependent on the quality of the training data for each appliance. Therefore, a common problem with NILM systems is that they may not generalize well to new environments where the appliances are unknown, which hinders their widespread adoption and more significant contributions to emerging smart grid services. The main goal of the presented research is to apply a domain adversarial neural network (DANN) approach to improve the generalization of NILM systems. The proposed semi-supervised algorithm utilizes both labeled and unlabeled data and was tested on data from publicly available REDD and UK-DALE datasets. The results show a 3% improvement in generalization performance on highly uncorrelated data, indicating the potential for real-world applications.