Sparse Regression Methods Research Articles

CR-Lasso: Robust cellwise regularized sparse regression

Cellwise contamination remains a challenging problem for data scientists, particularly in research fields that require the selection of sparse features. Traditional robust methods may not be feasible nor efficient in dealing with such contaminated datasets. A robust Lasso-type cellwise regularization procedure is proposed which is coined CR-Lasso, that performs feature selection in the presence of cellwise outliers by minimising a regression loss and cell deviation measure simultaneously. The evaluation of this approach involves simulation studies that compare its selection and prediction performance with several sparse regression methods. The results demonstrate that CR-Lasso is competitive within the considered settings. The effectiveness of the proposed method is further illustrated through an analysis of a bone mineral density dataset.

Kernel functions embed into the autoencoder to identify the sparse models of nonlinear dynamics

Numerous researches have shown that there are three main challenges in data-driven model identification methods: high-dimensional measurements, system complexity and unknown underlying dynamical properties. For most nonlinear dynamics, the feature space defined by the coefficients of their control equations is sparse. Therefore, sparse regression methods are used to learn the sparse coefficients of the control equations of nonlinear dynamics. However, this method strongly depends on the appropriate selection of the sparse basis vectors. In this essay, the autoencoder is combined with the sparse regression method to simultaneously identify the sparse coordinate and a parsimonious, interpretable and generalizable model of the specified system. It also integrates kernel functions to map the intractable measurements in the hidden space of the autoencoder into a linearly distinguishable kernel space, which kernelizes the candidate function library of the sparse identification of nonlinear dynamics (SINDy) model as the sparse dictionaries. Therefore, the flexible representation of neural networks, the simplicity of sparse regression methods and the implicit non-linear representation of kernel functions are consolidated in this article. To inspect the reliability of the proposed model in this paper, a set of nonlinear dynamics formulated by ordinary differential equations (ODEs), second-order trigonometric functions and partial differential equations (PDEs) are utilized as test cases. And the comparisons between the proposed model and other model identification methods illustrate that the performance of the former is the best.

Sparse regression system identification in two-phase flow metering

Multiphase flow metering is an open research field, with challenges encompassing sensor development and thermofluid models, for instance. Sparse regression is a modern system identification strategy able to provide simple models with Physical interpretation. This papers applies sparse identification for the assessment of gas velocity, gas fraction and superficial velocities of liquid and gas in experimental air–water horizontal slug flow across a Venturi tube and a twin-plane capacitive sensor. We show that this technique can improve measurement accuracies, as deviations for superficial velocities fall below 2.8% for liquid and 8.7% for gas. More importantly, the analysis of three data sets discusses practical concerns when applying sparse regression methods in metering, including the choice of basis functions, the degree of sparsity and overfitting. Overall, sparse identification is perceived as an adequate method to simultaneously generate a measurement model and correction of measurement biases in a specific measurement setup.

Identification of a soil water flow equation using a two-step data-driven method

Governing equations of soil water flow are traditionally derived from the conservation of mass and the Buckingham-Darcy law. However, due to uncertainties in many aspects, such as physical processes and constitutive relationships, a thorough understanding of soil water flow in a theory-driven way often poses many challenges. With the fast improvement of data acquisition ability, data-driven approaches have gained considerable scientific interest in discovering the governing equations of physical problems. However, most existing studies focused on linear or weakly nonlinear problems with constant coefficients in their partial differential equations. This study attempted to propose a modified two-step data-driven method to identify the strongly nonlinear equation of unsaturated flow. The Richardson-Richards equation and coefficients were jointly identified by hybridizing sparse regression and iterative smoother. The automatic differentiation scheme was introduced to reduce the risk of performance collapse from data noise common in sparse regression methods. We demonstrated the ability, and the challenge as well, of identifying the soil water flow governing equations in a data-driven way with the aid of a series of synthetic cases. Our results showed that the modified method was capable of accurately identifying the governing equation of unsaturated flow, whose performance was almost independent of prior knowledge of the equation form and constitutive relationships. If the underlying physical processes are poorly understood, then a more relaxed equation form was recommended. The reduction in the spatiotemporal resolution of sampling may cause higher-order terms to be difficult to derive accurately or some terms that were not originally present to appear in the identified equation. Although not completely eliminated, the detrimental effect of data noise on the identification of the governing equations can be greatly mitigated by introducing the automatic differentiation scheme.

Stochastic variational inference for scalable non-stationary Gaussian process regression

A natural extension to standard Gaussian process (GP) regression is the use of non-stationary Gaussian processes, an approach where the parameters of the covariance kernel are allowed to vary in time or space. The non-stationary GP is a flexible model that relaxes the strong prior assumption of standard GP regression, that the covariance properties of the inferred functions are constant across the input space. Non-stationary GPs typically model varying covariance kernel parameters as further lower-level GPs, thereby enabling sampling-based inference. However, due to the high computational costs and inherently sequential nature of MCMC sampling, these methods do not scale to large datasets. Here we develop a variational inference approach to fitting non-stationary GPs that combines sparse GP regression methods with a trajectory segmentation technique. Our method is scalable to large datasets containing potentially millions of data points. We demonstrate the effectiveness of our approach on both synthetic and real world datasets.

Learning sparse nonlinear dynamics via mixed-integer optimization

Discovering governing equations of complex dynamical systems directly from data is a central problem in scientific machine learning. In recent years, the sparse identification of nonlinear dynamics (SINDy) framework, powered by heuristic sparse regression methods, has become a dominant tool for learning parsimonious models. We propose an exact formulation of the SINDy problem using mixed-integer optimization (MIO-SINDy) to solve the sparsity constrained regression problem to provable optimality in seconds. On a large number of canonical ordinary and partial differential equations, we illustrate the dramatic improvement in our approach in accurate model discovery while being more sample efficient, robust to noise, and flexible in accommodating physical constraints.

Integrated Identification of the Nonlinear Autoregressive Models With Exogenous Inputs (NARX) for Engineering Systems Design

This brief presents a new framework for the identification of nonlinear autoregressive (AR) models with exogenous inputs (NARX) model for design (NARX-M-for-D), which represents NARX of engineering systems where the model coefficients are represented explicitly as a function of the physical parameters that can be adjusted for the system design. The framework is concerned with identifying a common structure of the NARX model which is shared by all design configurations, and with identifying the nonlinear static maps that link these design parameters with NARX coefficients. The problem of the common structure identification is solved via extended forward orthogonal regression, after which a joint regression problem is formulated to determine the explicit relationships between NARX coefficients and physical parameters for the system. Using sparse regression methods allows simultaneous detection of a compact structure of the NARX model and design parameter maps. The reduced structure improves model generalization in design parameter space which is instrumental for the application of the identified model in the system design. The performance of the framework is evaluated on a benchmark model and on the experimental data from dynamic testing of auxetic foams. An example of evaluating the output frequency response from the identified model demonstrates how the proposed framework can be used to assess dynamical properties of engineered systems in the design process.

Learning chaotic systems from noisy data via multi-step optimization and adaptive training.

A data-driven sparse identification method is developed to discover the underlying governing equations from noisy measurement data through the minimization of Multi-Step-Accumulation (MSA) in error. The method focuses on the multi-step model, while conventional sparse regression methods, such as the Sparse Identification of Nonlinear Dynamics method (SINDy), are one-step models. We adopt sparse representation and assume that the underlying equations involve only a small number of functions among possible candidates in a library. The new development in MSA is to use a multi-step model, i.e., predictions from an approximate evolution scheme based on initial points. Accordingly, the loss function comprises the total error at all time steps between the measured series and predicted series with the same initial point. This enables MSA to capture the dynamics directly from the noisy measurements, resisting the corruption of noise. By use of several numerical examples, we demonstrate the robustness and accuracy of the proposed MSA method, including a two-dimensional chaotic map, the logistic map, a two-dimensional damped oscillator, the Lorenz system, and a reduced order model of a self-sustaining process in turbulent shear flows. We also perform further studies under challenging conditions, such as noisy measurements, missing data, and large time step sizes. Furthermore, in order to resolve the difficulty of the nonlinear optimization, we suggest an adaptive training strategy, namely, by gradually increasing the length of time series for training. Higher prediction accuracy is achieved in an illustrative example of the chaotic map by the adaptive strategy.

A sparse regression approach for populating dark matter haloes and subhaloes with galaxies

ABSTRACT We use sparse regression methods (SRMs) to build accurate and explainable models that predict the stellar mass of central and satellite galaxies as a function of properties of their host dark matter haloes. SRMs are machine learning algorithms that provide a framework for modelling the governing equations of a system from data. In contrast with other machine learning algorithms, the solutions of SRM methods are simple and depend on a relatively small set of adjustable parameters. We collect data from 35 459 galaxies from the EAGLE simulation using 19 redshift slices between z = 0 and z = 4 to parametrize the mass evolution of the host haloes. Using an appropriate formulation of input parameters, our methodology can model satellite and central haloes using a single predictive model that achieves the same accuracy as when predicted separately. This allows us to remove the somewhat arbitrary distinction between those two galaxy types and model them based only on their halo growth history. Our models can accurately reproduce the total galaxy stellar mass function and the stellar mass-dependent galaxy correlation functions (ξ(r)) of EAGLE. We show that our SRM model predictions of ξ(r) is competitive with those from subhalo abundance matching and might be comparable to results from extremely randomized trees. We suggest SRM as an encouraging approach for populating the haloes of dark matter only simulations with galaxies and for generating mock catalogues that can be used to explore galaxy evolution or analyse forthcoming large-scale structure surveys.

Quality control, classification, and authentication of Iranian rice varieties using FT-IR spectroscopy and sparse chemometric methods

The authentication and classification of rice varieties have attracted much attention in food industry, in recent years. In this study, FTIR spectroscopy and sparse chemometric methods were used for discriminating different classes of Iranian rice samples and detecting adulteration in high-quality products. The sparse version of the linear discriminant analysis (sLDA) was used for developing interpretable classification models and detecting adulteration in rice samples. More than 400 samples from eight different classes of the most favorable Iranian rice varieties were used for developing sparse models. The performance of the sLDA model was compared with common chemometric methods, and the results revealed the superiority of the sLDA in terms of interpretability and prediction accuracy. Moreover, sparse multivariate regression methods including least absolute shrinkage and selection operator (lasso), ridge, and elastic net were used for quantitative analysis and fast assessment of rice adulteration. The overall accuracy for sLDA was 0.967 and the ranges of coefficient of determination (R2) for lasso models were 0.879–0.948, for the test sets. The results showed that the sparse chemometric methods combined with FTIR spectroscopy can be successfully applied as robust and interpretable tools for classification and adulteration detection of rice samples in food industry.

Robust physics discovery via supervised and unsupervised pattern recognition using the Euler Characteristic

Machine learning approaches have been widely used for discovering the underlying physics of dynamical systems from measured data. Existing approaches, however, still lack robustness, especially when the measured data contain a large level of noise. The lack of robustness is mainly attributed to the insufficient representativeness of used features. As a result, the intrinsic mechanism governing the observed system cannot be accurately identified. In this study, we propose a robust physics discovery method via pattern recognition. In this method, the Euler Characteristic (EC), an efficient topological descriptor for complex data, is used as the feature vector for characterizing the spatiotemporal data collected from dynamical systems. Unsupervised manifold learning and supervised classification results show that EC can be used to efficiently distinguish systems with different while similar governing models. We also demonstrate that the machine learning approaches using EC can improve the results of sparse regression methods of physics discovery without hard-thresholding or hyperparameter tuning.

A Priori Denoising Strategies for Sparse Identification of Nonlinear Dynamical Systems: A Comparative Study

Abstract In recent years, identification of nonlinear dynamical systems from data has become increasingly popular. Sparse regression approaches, such as Sparse Identification of Nonlinear Dynamics (SINDy), fostered the development of novel governing equation identification algorithms assuming the state variables are known {\it a priori} and the governing equations lend themselves to sparse, linear expansions in a (nonlinear) basis of the state variables. In the context of the identification of governing equations of nonlinear dynamical systems, one faces the problem of identifiability of model parameters when state measurements are corrupted by noise. Measurement noise affects the stability of the recovery process yielding incorrect sparsity patterns and inaccurate estimation of coefficients of the governing equations. In this work, we investigate and compare the performance of several local and global smoothing techniques to {\it a priori} denoise the state measurements and numerically estimate the state time-derivatives to improve the accuracy and robustness of two sparse regression methods to recover governing equations: Sequentially Thresholded Least Squares (STLS) and Weighted Basis Pursuit Denoising (WBPDN) algorithms. We empirically show that, in general, global methods, which use the entire measurement data set, outperform local methods, which employ a neighboring data subset around a local point. We additionally compare Generalized Cross Validation (GCV) and Pareto curve criteria as model selection techniques to automatically estimate near optimal tuning parameters, and conclude that Pareto curves yield better results. The performance of the denoising strategies and sparse regression methods is empirically evaluated through well-known benchmark problems of nonlinear dynamical systems.

Ideal algorithms in healthcare: Explainable, dynamic, precise, autonomous, fair, and reproducible.

Established guidelines describe minimum requirements for reporting algorithms in healthcare; it is equally important to objectify the characteristics of ideal algorithms that confer maximum potential benefits to patients, clinicians, and investigators. We propose a framework for ideal algorithms, including 6 desiderata: explainable (convey the relative importance of features in determining outputs), dynamic (capture temporal changes in physiologic signals and clinical events), precise (use high-resolution, multimodal data and aptly complex architecture), autonomous (learn with minimal supervision and execute without human input), fair (evaluate and mitigate implicit bias and social inequity), and reproducible (validated externally and prospectively and shared with academic communities). We present an ideal algorithms checklist and apply it to highly cited algorithms. Strategies and tools such as the predictive, descriptive, relevant (PDR) framework, the Standard Protocol Items: Recommendations for Interventional Trials-Artificial Intelligence (SPIRIT-AI) extension, sparse regression methods, and minimizing concept drift can help healthcare algorithms achieve these objectives, toward ideal algorithms in healthcare.

Uncertainty Analysis and Experimental Validation of Identifying the Governing Equation of an Oscillator Using Sparse Regression

In recent years, the rapid growth of computing technology has enabled identifying mathematical models for vibration systems using measurement data instead of domain knowledge. Within this category, the method Sparse Identification of Nonlinear Dynamical Systems (SINDy) shows potential for interpretable identification. Therefore, in this work, a procedure of system identification based on the SINDy framework is developed and validated on a single-mass oscillator. To estimate the parameters in the SINDy model, two sparse regression methods are discussed. Compared with the Least Squares method with Sequential Threshold (LSST), which is the original estimation method from SINDy, the Least Squares method Post-LASSO (LSPL) shows better performance in numerical Monte Carlo Simulations (MCSs) of a single-mass oscillator in terms of sparseness, convergence, identified eigenfrequency, and coefficient of determination. Furthermore, the developed method SINDy-LSPL was successfully implemented with real measurement data of a single-mass oscillator with known theoretical parameters. The identified parameters using a sweep signal as excitation are more consistent and accurate than those identified using impulse excitation. In both cases, there exists a dependency of the identified parameter on the excitation amplitude that should be investigated in further research.

A novel polynomial dimension decomposition method based on sparse Bayesian learning and Bayesian model averaging

The polynomial dimensional decomposition (PDD), an orthogonal polynomial-based metamodel, has received increasing attention in uncertainty quantification (UQ). Nevertheless, for complex high-dimensional problems, its computational burden may become unaffordable, which is usually called curse of dimensionality. To solve this problem, sparse regression methods can be considered to establish a sparse PDD model. However, when the design samples are limited, their computational accuracy may be low due to the enormous size of the polynomial bases. Aimed at this issue, we proposed a novel sparse PDD metamodel based on the Bayesian LASSO (least absolute shrinkage and selection operator) method and an adaptive candidate basis selection and model updating method (CBSMU). Firstly, to improve the CPU efficiency, an analytical Bayesian LASSO based on the sparse Bayesian learning is used for the regression analysis, which replaces the time-consuming Markov chain Monte Carlo sampling of the traditional method with the efficient iteration algorithm for calculating the posterior estimations. Then, to reduce the size of the polynomial bases, this study proposes the adaptive CBSMU for screening the significant candidate polynomial bases and updating the metamodel sequentially. The CBSMU can find out the candidate bases that contributes to improve the prediction accuracy in the view of Bayesian model averaging. Thus, during the process of the sparse PDD modeling, the size of candidate bases is relatively small, which facilitates to improve the final computational accuracy when the design samples are limited. We verify the proposed method using three high-dimensional numerical examples, and apply it to solve one complex high-dimensional engineering problem. The results show that the proposed method is more accurate for UQ than the two common methods with the same computational costs, and is well-suited for solving complex high-dimensional structural dynamic problem.

Bayesian factor-adjusted sparse regression

Many sparse regression methods rely on an assumption that the covariates are weakly correlated, which hardly holds in many economic and financial datasets. To relax this assumption, we model the strongly correlated covariates by a factor structure: strong correlations among covariates are modeled by common factors, while the remaining variations of covariates are modeled as idiosyncratic components. We then propose a factor-adjusted sparse regression model and develop a semi-Bayesian estimation method for it. Posterior contraction rate and model selection consistency are established by a non-asymptotic analysis. Experimental studies show that the proposed method outperforms its Lasso analogue, manifests insensitivity to overestimates of the number of common factors, pays a negligible price when covariates are uncorrelated, scales up well with increasing sample size, dimensionality and sparsity, and converges fast to the posterior distribution. An application to the U.S. bond risk premia lends further support to the proposed model and method.

Double/debiased machine learning for logistic partially linear model.

We propose double/debiased machine learning approaches to infer a parametric component of a logistic partially linear model. Our framework is based on a Neyman orthogonal score equation consisting of two nuisance models for the nonparametric component of the logistic model and conditional mean of the exposure with the control group. To estimate the nuisance models, we separately consider the use of high dimensional (HD) sparse regression and (nonparametric) machine learning (ML) methods. In the HD case, we derive certain moment equations to calibrate the first order bias of the nuisance models, which preserves the model double robustness property. In the ML case, we handle the nonlinearity of the logit link through a novel and easy-to-implement 'full model refitting' procedure. We evaluate our methods through simulation and apply them in assessing the effect of the emergency contraceptive pill on early gestation and new births based on a 2008 policy reform in Chile.

Performances of some high dimensional regression methods: sparse principal component regression

Principal component analysis (PCA) is widely used technique in data processing and dimensionality reduction, but it has some drawbacks since each principal component is a linear combination of explanatory variables. As an alternative, the sparse PCA (SPCA) is a very appealing method which produces principal components with sparse loadings. On the other hand, combining PCA on explanatory variables with least squares regression yields to principal component regression (PCR). In PCR, the components are obtained using only explanatory variables, not considered the effect of the dependent variable. Considering the dependent variable, the sparse PCR (SPCR) enables to obtain sparse principal component loadings. But the main drawback of it is the computational cost. Taking into consideration the general structure of PCR, we combine (S)PCA with some sparse regression methods and compared with the classical PCR and last introduced method, SPCR. Extensive simulation studies and real data examples are implemented to show their performances. The results are supported by a reasonable computation time study.

Predicting lecithin concentration from differential mobility spectrometry measurements with linear regression models and neural networks

Differential mobility spectrometry (DMS) analysis of electrosurgical smoke can be used to distinguish cancerous and healthy tissues. Mass spectrometry studies of surgical smoke have revealed phospholipids as the key compounds enabling this discrimination. Lecithin is a mixture of phospholipids encountered in tissues. We hypothesized that DMS is capable of detecting and quantifying lecithin from water solution in headspace chamber, paving way for analysis of surgical smoke. We measured different lecithin concentrations in a biologically relevant range considering healthy and cancerous tissues with DMS and trained regression models to predict the analyte concentration. The models were internally cross-validated and externally validated. The best cross-validation results were obtained with convolutional neural networks, with root mean square error (RMSE) = 0.38 mg/ml. This is the first demonstration of estimation of analyte concentration from DMS measurements with neural networks. The best external validation results were acquired with sparse linear regression methods, with RMSE varying from 0.40 mg/ml to 0.41 mg/ml. The results demonstrate that DMS is sufficiently sensitive to detect biologically relevant changes in phospholipid concentration, potentially explaining its ability to detect cancerous tissue. In the future, we aim to reproduce the results by using surgical smoke as the medium. In this scenario, the complex background of surgical smoke will be the main challenge to overcome. Predicting concentration with neural networks also lays the foundation for wider analytical usage of DMS.

Cancer Diagnosis and Disease Gene Identification via Statistical Machine Learning

Diagnosing cancer and identifying the disease gene by using DNA microarray gene expression data are the hot topics in current bioinformatics. This paper is devoted to the latest development in cancer diagnosis and gene selection via statistical machine learning. A support vector machine is firstly introduced for the binary cancer diagnosis. Then, 1-norm support vector machine, doubly regularized support vector machine, adaptive huberized support vector machine and other extensions are presented to improve the performance of gene selection. Lasso, elastic net, partly adaptive elastic net, group lasso, sparse group lasso, adaptive sparse group lasso and other sparse regression methods are also introduced for performing simultaneous binary cancer classification and gene selection. In addition to introducing three strategies for reducing multiclass to binary, methods of directly considering all classes of data in a learning model (multi_class support vector, sparse multinomial regression, adaptive multinomial regression and so on) are presented for performing multiple cancer diagnosis. Limitations and promising directions are also discussed.