Automatic Model Selection Research Articles

Dynamic robustness evaluation for automated model selection in operation

Context:The increasing use of artificial neural network (ANN) classifiers in systems, especially safety-critical systems (SCSs), requires ensuring their robustness against out-of-distribution (OOD) shifts in operation, which are changes in the underlying data distribution from the data training the classifier. However, measuring the robustness of classifiers in operation with only unlabeled data is challenging. Additionally, machine learning engineers may need to compare different models or versions of the same model and switch to an optimal version based on their robustness. Objective:This paper explores the problem of dynamic robustness evaluation for automated model selection. We aim to find efficient and effective metrics for evaluating and comparing the robustness of multiple ANN classifiers using unlabeled operational data. Methods:To quantitatively measure the differences between the model outputs and assess robustness under OOD shifts using unlabeled data, we choose distance-based metrics. An empirical comparison of five such metrics, suitable for higher-dimensional data like images, is performed. The selected metrics include Wasserstein distance (WD), maximum mean discrepancy (MMD), Hellinger distance (HL), Kolmogorov–Smirnov statistic (KS), and Kullback–Leibler divergence (KL), known for their efficacy in quantifying distribution differences. We evaluate these metrics on 20 state-of-the-art models (ten CIFAR10-based models, five CIFAR100-based models, and five ImageNet-based models) from a widely used robustness benchmark (RobustBench) using data perturbed with various types and magnitudes of corruptions to mimic real-world OOD shifts. Results:Our findings reveal that the WD metric outperforms others when ranking multiple ANN models for CIFAR10- and CIFAR100-based models, while the KS metric demonstrates superior performance for ImageNet-based models. MMD can be used as a reliable second option for both datasets. Conclusion:This study highlights the effectiveness of distance-based metrics in ranking models’ robustness for automated model selection. It also emphasizes the significance of advancing research in dynamic robustness evaluation.

Automated model selection for multivariate anomaly detection in manufacturing systems

AbstractAs machine learning is widely applied to improve the efficiency and effectiveness of manufacturing systems, the automated selection of appropriate algorithms and hyperparameters becomes increasingly important. This paper presents a model selection approach to multivariate anomaly detection for applications in manufacturing systems using a multi-output regression-based meta-learning method. The proposed method exploits the capabilities of meta-learning to explore and learn the intricate relationships within multivariate data sets in order to select the best anomaly detection model. It also facilitates the construction of an ensemble of algorithms with dynamically assigned weights based on their respective performance levels. In addition to the framework, new meta-features for the application domain are presented and evaluated. Experiments show the proposed method can be successfully applied to achieve significantly better results than benchmark approaches. This enables an automated selection of algorithms that can be used for enhanced anomaly detection under changing operating conditions.

Learning macroscopic equations of motion from dissipative particle dynamics simulations of fluids

Macroscopic descriptions of both natural and engineered materials usually include a number of phenomenological parameters that have to be estimated from experiments or large-scale microscopic simulations. When dealing with advanced complex materials, these descriptions are sometimes not a priori available or not even known. Using sparsity-promoting techniques one can extract macroscopic dynamic models directly from particle-based simulations. In this work, we showcase such an approach on a simple fluid and test its robustness. We introduce a novel measure for automatic macroscopic model selection that combines stability and accuracy of a model. Using this measure and employing only a few physics-based assumptions, we are able to infer both the mass continuity equation and an equation for the conservation of linear momentum. Moreover, the extracted phenomenological and non-phenomenological parameters agree well with their numerically measured values and the well-known semi-empirical estimates. The presented model selection framework can be applied to simulations or experimental data of more complex systems, described in general by a rich set of coupled nonlinear macroscopic equations.

Split-and-merge model selection of mixtures of Gaussian processes with RJMCMC

The mixture of Gaussian processes is a powerful statistical learning model that can be effectively applied to curve clustering and prediction. However, the corresponding model selection problem, that is, selecting an appropriate number of components in the mixture, is rather difficult to solve. In our previous work, we established the split-and-merge automatic model selection algorithm for mixtures of Gaussian processes along the output space under the framework of Reversible Jump Markov Chain Monte Carlo (RJMCMC), which can not only determine the number of actual Gaussian processes but also dynamically adjust the Gaussian process components to avoid dependence on parameter initialization and initial partitioning of the dataset during the parameter learning on a given dataset. In this study, we propose two algorithms: Penalized Likelihood RJMCMC and Penalized Prior RJMCMC. The former integrates a penalized term into the likelihood, while the latter incorporates a penalized term into the prior and operates within the full Bayesian inference framework, both aiming to focus more sharply on determining the number of components in the convergence process. Furthermore, we prove the geometric ergodicity of the RJMCMC algorithm for the mixture of Gaussian processes model, ensuring convergence of the posterior distribution with sufficient iterations. The experimental results further demonstrate the robustness of our PP-RJMCMC algorithm in model selection, showing superior performance compared to traditional approaches in curve classification and clustering. Additionally, the prediction performance is comparable to the EM algorithm. Although not directly explored in this study, the RJMCMC results can be used to initialize the EM algorithm, which could potentially improve prediction accuracy and accelerate computation.

Towards Automated Model Selection for Wind Speed and Solar Irradiance Forecasting.

Given the recent increase in demand for electricity, it is necessary for renewable energy sources (RESs) to be widely integrated into power networks, with the two most commonly adopted alternatives being solar and wind power. Nonetheless, there is a significant amount of variation in wind speed and solar irradiance, on both a seasonal and a daily basis, an issue that, in turn, causes a large degree of variation in the amount of solar and wind energy produced. Therefore, RES technology integration into electricity networks is challenging. Accurate forecasting of solar irradiance and wind speed is crucial for the efficient operation of renewable energy power plants, guaranteeing the electricity supply at the most competitive price and preserving the dependability and security of electrical networks. In this research, a variety of different models were evaluated to predict medium-term (24 h ahead) wind speed and solar irradiance based on real-time measurement data relevant to the island of Crete, Greece. Illustrating several preprocessing steps and exploring a collection of "classical" and deep learning algorithms, this analysis highlights their conceptual design and rationale as time series predictors. Concluding the analysis, it discusses the importance of the "features" (intended as "time steps"), showing how it is possible to pinpoint the specific time of the day that most influences the forecast. Aside from producing the most accurate model for the case under examination, the necessity of performing extensive model searches in similar studies is highlighted by the current work.

Automated selection of nanoparticle models for small-angle X-ray scattering data analysis using machine learning.

Small-angle X-ray scattering (SAXS) is widely used to analyze the shape and size of nanoparticles in solution. A multitude of models, describing the SAXS intensity resulting from nanoparticles of various shapes, have been developed by the scientific community and are used for data analysis. Choosing the optimal model is a crucial step in data analysis, which can be difficult and time-consuming, especially for non-expert users. An algorithm is proposed, based on machine learning, representation learning and SAXS-specific preprocessing methods, which instantly selects the nanoparticle model best suited to describe SAXS data. The different algorithms compared are trained and evaluated on a simulated database. This database includes 75 000 scattering spectra from nine nanoparticle models, and realistically simulates two distinct device configurations. It will be made freely available to serve as a basis of comparison for future work. Deploying a universal solution for automatic nanoparticle model selection is a challenge made more difficult by the diversity of SAXS instruments and their flexible settings. The poor transferability of classification rules learned on one device configuration to another is highlighted. It is shown that training on several device configurations enables the algorithm to be generalized, without degrading performance compared with configuration-specific training. Finally, the classification algorithm is evaluated on a real data set obtained by performing SAXS experiments on nanoparticles for each of the instrumental configurations, which have been characterized by transmission electron microscopy. This data set, although very limited, allows estimation of the transferability of the classification rules learned on simulated data to real data.

Forecasting seasonal influenza activity in Canada-Comparing seasonal Auto-Regressive integrated moving average and artificial neural network approaches for public health preparedness.

Public health preparedness is based on timely and accurate information. Time series forecasting using disease surveillance data is an important aspect of preparedness. This study compared two approaches of time series forecasting: seasonal auto-regressive integrated moving average (SARIMA) modelling and the artificial neural network (ANN) algorithm. The goal was to model weekly seasonal influenza activity in Canada using SARIMA and compares its predictive accuracy, based on root mean square prediction error (RMSE) and mean absolute prediction error (MAE), to that of an ANN. An initial SARIMA model was fit using automated model selection by minimizing the Akaike information criterion (AIC). Further inspection of the autocorrelation function and partial autocorrelation function led to 'manual' model improvements. ANNs were trained iteratively, using an automated process to minimize the RMSE and MAE. A total of 378, 462 cases of influenza was reported in Canada from the 2010-2011 influenza season to the end of the 2019-2020 influenza season, with an average yearly incidence risk of 20.02 per 100,000 population. Automated SARIMA modelling was the better method in terms of forecasting accuracy (per RMSE and MAE). However, the ANN correctly predicted the peak week of disease incidence while the other models did not. Both the ANN and SARIMA models have shown to be capable tools in forecasting seasonal influenza activity in Canada. It was shown that applying both in tandem is beneficial, SARIMA better forecasted overall incidence while ANN correctly predicted the peak week.

Automated Model Selection Using Bayesian Optimization and the Asynchronous Successive Halving Algorithm for Predicting Daily Minimum and Maximum Temperatures

This study addresses the crucial role of temperature forecasting, particularly in agricultural contexts, where daily maximum (Tmax) and minimum (Tmin) temperatures significantly impact crop growth and irrigation planning. While machine learning (ML) models offer a promising avenue for temperature forecasts, the challenge lies in efficiently training multiple models and optimizing their parameters. This research addresses a research gap by proposing advanced ML algorithms for multi-step-ahead Tmax and Tmin forecasting across various weather stations in Bangladesh. The study employs Bayesian optimization and the asynchronous successive halving algorithm (ASHA) to automatically select top-performing ML models by tuning hyperparameters. While both the Bayesian and ASHA optimizations yield satisfactory results, ASHA requires less computational time for convergence. Notably, different top-performing models emerge for Tmax and Tmin across various forecast horizons. The evaluation metrics on the test dataset confirm higher accuracy, efficiency coefficients, and agreement indices, along with lower error values for both Tmax and Tmin forecasts at different weather stations. Notably, the forecasting accuracy decreases with longer horizons, emphasizing the superiority of one-step-ahead predictions. The automated model selection approach using Bayesian and ASHA optimization algorithms proves promising for enhancing the precision of multi-step-ahead temperature forecasting, with potential applications in diverse geographical locations.

Automatic selection model to identify neurodegenerative diseases.

This study evaluates machine learning algorithms' effectiveness in classifying Parkinson's disease and Huntington's disease based on biomarker data obtained non-invasively from patients and healthy controls. Datasets containing biomarker data (x, y, and z values of accelerometers) from sensors were collected from Parkinson's disease, Huntington's disease patients, and healthy controls. An automatic selection model method was implemented for disease classification, using a unique Mexican database of human gait biomarkers, which we consider the only one of its kind. Random forest, random subspace method, and K-star algorithms were employed, with parameters optimized through an automated model selection. The study achieved a 0.893 precision rate for Parkinson's disease and Huntington's disease using the random subspace method. The findings underscore the potential of machine learning techniques in medical diagnosis, particularly in neurological disorders. The automatic selection model method demonstrated efficacy in classifying Parkinson's disease and Huntington's disease based on non-invasive biomarker data. This research contributes to advancing non-invasive diagnostic approaches in neurological disorders, highlighting the significance of machine learning in healthcare.

Comparison between inverse-probability weighting and multiple imputation in Cox model with missing failure subtype.

Identifying and distinguishing risk factors for heterogeneous disease subtypes has been of great interest. However, missingness in disease subtypes is a common problem in those data analyses. Several methods have been proposed to deal with the missing data, including complete-case analysis, inverse-probability weighting, and multiple imputation. Although extant literature has compared these methods in missing problems, none has focused on the competing risk setting. In this paper, we discuss the assumptions required when complete-case analysis, inverse-probability weighting, and multiple imputation are used to deal with the missing failure subtype problem, focusing on how to implement these methods under various realistic scenarios in competing risk settings. Besides, we compare these three methods regarding their biases, efficiency, and robustness to model misspecifications using simulation studies. Our results show that complete-case analysis can be seriously biased when the missing completely at random assumption does not hold. Inverse-probability weighting and multiple imputation estimators are valid when we correctly specify the corresponding models for missingness and for imputation, and multiple imputation typically shows higher efficiency than inverse-probability weighting. However, in real-world studies, building imputation models for the missing subtypes can be more challenging than building missingness models. In that case, inverse-probability weighting could be preferred for its easy usage. We also propose two automated model selection procedures and demonstrate their usage in a study of the association between smoking and colorectal cancer subtypes in the Nurses' Health Study and Health Professional Follow-Up Study.

Temporal classification of short time series data

MotivationWithin the frame of their genetic capacity, organisms are able to modify their molecular state to cope with changing environmental conditions or induced genetic disposition. As high throughput methods are becoming increasingly affordable, time series analysis techniques are applied frequently to study the complex dynamic interplay between genes, proteins, and metabolites at the physiological and molecular level. Common analysis approaches fail to simultaneously include (i) information about the replicate variance and (ii) the limited number of responses/shapes that a biological system is typically able to take.ResultsWe present a novel approach to model and classify short time series signals, conceptually based on a classical time series analysis, where the dependency of the consecutive time points is exploited. Constrained spline regression with automated model selection separates between noise and signal under the assumption that highly frequent changes are less likely to occur, simultaneously preserving information about the detected variance. This enables a more precise representation of the measured information and improves temporal classification in order to identify biologically interpretable correlations among the data.Availability and implementationAn open source F# implementation of the presented method and documentation of its usage is freely available in the TempClass repository, https://github.com/CSBiology/TempClass [58].

Iterative Feedback Tuning with automated reference model selection

Iterative Feedback Tuning (IFT) is a direct, data-driven control technique, that relies on a reference model to capture the desired behavior of the unknown system. The choice of this hyper-parameter is particularly critical, as it potentially jeopardizes performance and even closed-loop stability. This paper aims to explore the suitability of three search methods (grid search, random search, and successive halving) to automatically tune the reference model from data based on a set of user-defined, soft specifications on the desired closed-loop behavior. To compare the three methods and demonstrate their effectiveness, we consider a benchmark simulation case study on the control of a mass-spring-damper system. From our results, successive halving turns out to be the most efficient way to run IFT with automatic reference model selection on a finite budget of time for data collection.

Automatic Initialization and Model Selection for Li-ion Battery Impedance Identification in the Frequency Domain

Electrochemical Impedance Spectroscopy (EIS) is a useful tool for selecting a pertinent Equivalent Circuit Model (ECM) of a Li-ion battery. Impedance model is designed to describe low, middle and high frequency electrochemical processes involved. When considering low frequency restricted in the Warburg zone, diffusion impedance is modeled thanks to a Constant Phase Element (CPE) which behaves as a fractional integrator of order n close to 0.5. Phenomena observed in middle frequency are described using specific circuits called Zarc which consist in connecting a CPE in parallel with a resistor. Therefore, the global impedance model is characterized by non integer order operators and parameters can be estimated by a Complex Non linear Least Squares (CNLS) algorithm which requires a proper initialization in order to guarantee the convergence to a global optimum. The paper presents a method to analyze EIS data measurements in order to select automatically the number of middle frequency Zarc circuits required (one or two) and to initialize properly the CNLS algorithm. The method is validated using experimental open source EIS data.

A Hybrid Bearing Prognostic Method With Fault Diagnosis and Model Fusion

Accurate bearing fault diagnosis and prognosis (FDP) is critical for optimal maintenance schedules, safety and reliability. The existing methods face some problems and challenges in detecting the starting time for prognosis and using a single model to describe fault dynamics. With these motivations, this paper presents a hybrid bearing FDP framework with fault detection and automatic fault model selection. In the proposed approach, convolutional neural network is used to detect fault and select the appropriate fault dynamic model. To improve the performance, power spectrum of vibration signals are fused with operating conditions to built information maps for fault detection and model selection. After a fault is detected, a Bayesian method is triggered to estimate the fault state and predict the remaining useful life. In the prognosis, Dempster-Shafer theory is employed to fuse prediction results from different models if necessary. The proposed approach is verified with bearings under different operating conditions. Experimental results and comparison studies verify the high accuracy and efficiency of the proposed method.

Enhancing Object Detection in Remote Sensing: A Hybrid YOLOv7 and Transformer Approach with Automatic Model Selection

In the remote sensing field, object detection holds immense value for applications such as land use classification, disaster monitoring, and infrastructure planning, where accurate and efficient identification of objects within images is essential for informed decision making. However, achieving object localization with high precision can be challenging even if minor errors exist at the pixel level, which can significantly impact the ground distance measurements. To address this critical challenge, our research introduces an innovative hybrid approach that combines the capabilities of the You Only Look Once version 7 (YOLOv7) and DEtection TRansformer (DETR) algorithms. By bridging the gap between local receptive field and global context, our approach not only enhances overall object detection accuracy, but also promotes precise object localization, a key requirement in the field of remote sensing. Furthermore, a key advantage of our approach is the introduction of an automatic selection module which serves as an intelligent decision-making component. This module optimizes the selection process between YOLOv7 and DETR, and further improves object detection accuracy. Finally, we validate the improved performance of our new hybrid approach through empirical experimentation, and thus confirm its contribution to the field of target recognition and detection in remote sensing images.

Structured model selection via optimization

Automated model selection is an important application in science and engineering. In this work, we develop a learning approach for identifying structured dynamical systems from undersampled and noisy spatiotemporal data. The learning is performed by a sparse least-squares fitting over a large set of candidate functions via a nonconvex sparse optimization solved by the alternating direction method of multipliers. We show that if the set of candidate functions forms a structured random sampling matrix of a bounded orthogonal system, the recovery is stable and the error is bounded. The learning approach is validated on synthetic data generated by the viscous Burgers’ equation and two reaction–diffusion equations. The computational results demonstrate the theoretical guarantees of success and the efficiency with respect to the number of candidate functions.

The generalized hyperbolic family and automatic model selection through the multiple‐choiceLASSO

AbstractWe revisit the generalized hyperbolic (GH) distribution and its nested models. These include widely used parametric choices like the multivariate normal, skew‐, Laplace, and several others. We also introduce the multiple‐choice LASSO, a novel penalized method for choosing among alternative constraints on the same parameter. A hierarchical multiple‐choice Least Absolute Shrinkage and Selection Operator (LASSO) penalized likelihood is optimized to perform simultaneous model selection and inference within the GH family. We illustrate our approach through a simulation study and a real data example. The methodology proposed in this paper has been implemented in R functions which are available as supplementary material.

Semi-Supervised Classification of Malware Families Under Extreme Class Imbalance via Hierarchical Non-Negative Matrix Factorization with Automatic Model Selection

Identification of the family to which a malware specimen belongs is essential in understanding the behavior of the malware and developing mitigation strategies. Solutions proposed by prior work, however, are often not practicable due to the lack of realistic evaluation factors. These factors include learning under class imbalance, the ability to identify new malware, and the cost of production-quality labeled data. In practice, deployed models face prominent, rare, and new malware families. At the same time, obtaining a large quantity of up-to-date labeled malware for training a model can be expensive. In this article, we address these problems and propose a novel hierarchical semi-supervised algorithm, which we call the HNMFk Classifier , that can be used in the early stages of the malware family labeling process. Our method is based on non-negative matrix factorization with automatic model selection, that is, with an estimation of the number of clusters. With HNMFk Classifier , we exploit the hierarchical structure of the malware data together with a semi-supervised setup, which enables us to classify malware families under conditions of extreme class imbalance. Our solution can perform abstaining predictions, or rejection option, which yields promising results in the identification of novel malware families and helps with maintaining the performance of the model when a low quantity of labeled data is used. We perform bulk classification of nearly 2,900 both rare and prominent malware families, through static analysis, using nearly 388,000 samples from the EMBER-2018 corpus. In our experiments, we surpass both supervised and semi-supervised baseline models with an F1 score of 0.80.

Strategies for automatic constitutive model selection and recommendation

The efficient development of high-quality metal products requires realistic numerical simulations prior to manufacturing. The choice of the constitutive model significantly influences material behaviour prediction and manufacturing process simulation. Numerous models exist to describe mechanical phenomena, making model selection challenging and prone to errors. This leads to costly delays in manufacturing. To address this issue, an automated material constitutive model selection and recommendation tool is essential. This research aims to develop a systematic strategy in three steps: analysis of variance (ANOVA), identifiability analysis, and identification quality. The approach is validated through simulations using a known material, including a hole expansion test and two strain heterogeneous tests for calibration. Step one establishes a ranking for model importance, aiding decision-making and accurate parameter calibration. Step two excludes models with inadequate identifiability and sensitivity measures. Step three confirms the methodology by showing improvements in calibration.

Distributed out-of-memory NMF on CPU/GPU architectures

AbstractWe propose an efficient distributed out-of-memory implementation of the non-negative matrix factorization (NMF) algorithm for heterogeneous high-performance-computing systems. The proposed implementation is based on prior work on NMFk, which can perform automatic model selection and extract latent variables and patterns from data. In this work, we extend NMFk by adding support for dense and sparse matrix operation on multi-node, multi-GPU systems. The resulting algorithm is optimized for out-of-memory problems where the memory required to factorize a given matrix is greater than the available GPU memory. Memory complexity is reduced by batching/tiling strategies, and sparse and dense matrix operations are significantly accelerated with GPU cores (or tensor cores when available). Input/output latency associated with batch copies between host and device is hidden using CUDA streams to overlap data transfers and compute asynchronously, and latency associated with collective communications (both intra-node and inter-node) is reduced using optimized NVIDIA Collective Communication Library (NCCL) based communicators. Benchmark results show significant improvement, from 32X to 76x speedup, with the new implementation using GPUs over the CPU-based NMFk. Good weak scaling was demonstrated on up to 4096 multi-GPU cluster nodes with approximately 25,000 GPUs when decomposing a dense 340 Terabyte-size matrix and an 11 Exabyte-size sparse matrix of density $$10^{-6}$$ 10 - 6 .