Sparse Regression Problem Research Articles

Sparse regression for discovery of constitutive models from oscillatory shear measurements

We propose sparse regression as an alternative to neural networks for the discovery of parsimonious constitutive models (CMs) from oscillatory shear experiments. Symmetry and frame invariance are strictly imposed by using tensor basis functions to isolate and describe unknown nonlinear terms in the CMs. We generate synthetic experimental data using the Giesekus and Phan-Thien Tanner CMs and consider two different scenarios. In the complete information scenario, we assume that the shear stress, along with the first and second normal stress differences, is measured. This leads to a sparse linear regression problem that can be solved efficiently using l1 regularization. In the partial information scenario, we assume that only shear stress data are available. This leads to a more challenging sparse nonlinear regression problem, for which we propose a greedy two-stage algorithm. In both scenarios, the proposed methods fit and interpolate the training data remarkably well. Predictions of the inferred CMs extrapolate satisfactorily beyond the range of training data for oscillatory shear. They also extrapolate reasonably well to flow conditions like startup of steady and uniaxial extension that are not used in the identification of CMs. We discuss ramifications for experimental design, potential algorithmic improvements, and implications of the non-uniqueness of CMs inferred from partial information.

A Network‐Constrain Weibull AFT Model for Biomarkers Discovery

ABSTRACTWe propose AFTNet, a novel network‐constraint survival analysis method based on the Weibull accelerated failure time (AFT) model solved by a penalized likelihood approach for variable selection and estimation. When using the log‐linear representation, the inference problem becomes a structured sparse regression problem for which we explicitly incorporate the correlation patterns among predictors using a double penalty that promotes both sparsity and grouping effect. Moreover, we establish the theoretical consistency for the AFTNet estimator and present an efficient iterative computational algorithm based on the proximal gradient descent method. Finally, we evaluate AFTNet performance both on synthetic and real data examples.

Nonlinear system identification via sparse Bayesian regression based on collaborative neurodynamic optimization

Abstract Sparse identification of nonlinear dynamics is a popular approach to system identification. In this approach system identification is reformulated as a sparse regression problem, and the use of a good sparse regression method is crucial. Sparse Bayesian learning based on collaborative neurodynamic optimization is a recent method that consistently produces high-quality solutions. In this article, we extensively assess how this method performs for ordinary differential equation identification. We find that it works very well compared with sparse regression algorithms currently used for this task in terms of the tradeoff between the approximation accuracy and the complexity of the identified system. We also propose a way to substantially reduce the computational complexity of this algorithm compared with its original implementation, thus making it even more practical.

Fast Sparse Discriminative K-Means for Unsupervised Feature Selection.

Embedded feature selection approach guides subsequent projection matrix (selection matrix) learning through the acquisition of pseudolabel matrix to conduct feature selection tasks. Yet the continuous pseudolabel matrix learned from relaxed problem based on spectral analysis deviates from reality to some extent. To cope with this issue, we design an efficient feature selection framework inspired by classical least-squares regression (LSR) and discriminative K-means (DisK-means), which is called the fast sparse discriminative K-means (FSDK) for the feature selection method. First, the weighted pseudolabel matrix with discrete trait is introduced to avoid trivial solution from unsupervised LSR. On this condition, any constraint imposed into pseudolabel matrix and selection matrix is dispensable, which is significantly beneficial to simplify the combinational optimization problem. Second, the l2,p -norm regularizer is introduced to satisfy the row sparsity of selection matrix with flexible p . Consequently, the proposed FSDK model can be treated as a novel feature selection framework integrated from the DisK-means algorithm and l2,p -norm regularizer to optimize the sparse regression problem. Moreover, our model is linearly correlated with the number of samples, which is speedy to handle the large-scale data. Comprehensive tests on various data terminally illuminate the effectiveness and efficiency of FSDK.

Sparsity-Preserved Pareto Optimization for Subset Selection

Subset selection that selects a limited number of variables optimizing many given criteria, is a fundamental problem with various applications such as sparse regression and unsupervised feature selection. Among the existing algorithms for subset selection, evolutionary algorithms (EAs) have achieved great performance in obtaining the subsets with high quality. However, as the selected subsets are sparse, i.e., most variables of these solutions are zero, most existing EAs for subset selection problems pose great challenges to find the optimal solutions effectively and efficiently, especially when the size of original set is large. To this end, we propose the SPESS approach which is a sparsity preserved evolutionary algorithm for subset selection. To be specific, we design two sparsity preserved operators (i.e., sparsity preserved crossover operator and sparsity preserved mutation operator), which can be used in SPESS to ensure the sparsity of the generated solutions in the process of Pareto optimization. The main advantages of the proposed sparsity preserved operators are that they can obtain high quality solutions and improve the search efficiency of the algorithm. According to the experimental results on 12 sparse regression problems and 12 unsupervised feature selection problems, the proposed SPESS is superior over the state-of-the-arts in solving subset selection tasks and the effectiveness of the proposed two sparsity preserved operators is also verified.

Sparse Approximate Pseudoinverse Preconditioning for Sparse Supervised Learning Problems with More Features than Samples

In this paper, we propose a new technique for handling a class of sparse supervised learning problems, specifically focusing on regression, binary, and multi-class classification problems. Sparse regression problems are associated with datasets including both arithmetic and categorical features and by encoding the dataset leads to an underdetermined sparse linear system. Furthermore, logistic regression can be used for both sparse binary and multi-class classification problems, solving an underdetermined sparse linear system. A new technique called Sparse Approximate Pseudoinverse Preconditioning (SAPP), namely the Explicit Preconditioned Conjugate Gradient for Normal Equations (EPCGNE) method based on Generic Approximate Sparse Pseudoinverse matrices is introduced for solving underdetermined sparse least square problems. Numerical experiments were carried out demonstrating a significant improvement of the performance metrics for the proposed SAPP scheme compared to other learners.

Conditioning of random Fourier feature matrices: double descent and generalization error

Abstract We provide high-probability bounds on the condition number of random feature matrices. In particular, we show that if the complexity ratio $N/m$, where $N$ is the number of neurons and $m$ is the number of data samples, scales like $\log ^{-1}(N)$ or $\log (m)$, then the random feature matrix is well-conditioned. This result holds without the need of regularization and relies on establishing various concentration bounds between dependent components of the random feature matrix. Additionally, we derive bounds on the restricted isometry constant of the random feature matrix. We also derive an upper bound for the risk associated with regression problems using a random feature matrix. This upper bound exhibits the double descent phenomenon and indicates that this is an effect of the double descent behaviour of the condition number. The risk bounds include the underparameterized setting using the least squares problem and the overparameterized setting where using either the minimum norm interpolation problem or a sparse regression problem. For the noiseless least squares or sparse regression cases, we show that the risk decreases as $m$ and $N$ increase. The risk bound matches the optimal scaling in the literature and the constants in our results are explicit and independent of the dimension of the data.

Estimation of multiple networks with common structures in heterogeneous subgroups

Network estimation has been a critical component of high-dimensional data analysis and can provide an understanding of the underlying complex dependence structures. Among the existing studies, Gaussian graphical models have been highly popular. However, they still have limitations due to the homogeneous distribution assumption and the fact that they are only applicable to small-scale data. For example, cancers have various levels of unknown heterogeneity, and biological networks, which include thousands of molecular components, often differ across subgroups while also sharing some commonalities. In this article, we propose a new joint estimation approach for multiple networks with unknown sample heterogeneity, by decomposing the Gaussian graphical model (GGM) into a collection of sparse regression problems. A reparameterization technique and a composite minimax concave penalty are introduced to effectively accommodate the specific and common information across the networks of multiple subgroups, making the proposed estimator significantly advancing from the existing heterogeneity network analysis based on the regularized likelihood of GGM directly and enjoying scale-invariant, tuning-insensitive, and optimization convexity properties. The proposed analysis can be effectively realized using parallel computing. The estimation and selection consistency properties are rigorously established. The proposed approach allows the theoretical studies to focus on independent network estimation only and has the significant advantage of being both theoretically and computationally applicable to large-scale data. Extensive numerical experiments with simulated data and the TCGA breast cancer data demonstrate the prominent performance of the proposed approach in both subgroup and network identifications.

Adaptive integral alternating minimization method for robust learning of nonlinear dynamical systems from highly corrupted data

This paper proposes an adaptive integral alternating minimization method (AIAMM) for learning nonlinear dynamical systems using highly corrupted measured data. This approach selects and identifies the system directly from noisy data using the integral model, encompassing unknown sparse coefficients, initial values, and outlier noisy data within the learning problem. It is defined as a sparse robust linear regression problem. An adaptive threshold parameter selection method is proposed to constrain model fitting errors and select appropriate threshold parameters for sparsity. The robustness and accuracy of the proposed AIAMM are demonstrated through several numerical experiments on typical nonlinear dynamical systems, including the van der Pol oscillator, Mathieu oscillator, Lorenz system, and 5D self-exciting homopolar disc dynamo. The proposed method is also compared to several advanced methods for sparse recovery, with the results indicating that the AIAMM demonstrates superior performance in processing highly corrupted data.

Multi-Objective Adaptive Optimization Model Predictive Control: Decreasing Carbon Emissions from a Zinc Oxide Rotary Kiln

The zinc oxide rotary kiln, as an essential piece of equipment in the zinc smelting industrial process, is presenting new challenges in process control. China’s strategy of achieving a carbon peak and carbon neutrality is putting new demands on the industry, including green production and the use of fewer resources; thus, traditional stability control is no longer suitable for multi-objective control tasks. Although researchers have revealed the principle of the rotary kiln and set up computational fluid dynamics (CFD) simulation models to study its dynamics, these models cannot be directly applied to process control due to their high computational complexity. To address these issues, this paper proposes a multi-objective adaptive optimization model predictive control (MAO-MPC) method based on sparse identification. More specifically, with a large amount of data collected from a CFD model, a sparse regression problem is first formulated and solved to obtain a reduction model. Then, a two-layered control framework including real-time optimization (RTO) and model predictive control (MPC) is designed. In the RTO layer, an optimization problem with the goal of achieving optimal operation performance and the lowest possible resource consumption is set up. By solving the optimization problem in real time, a suitable setting value is sent to the MPC layer to ensure that the zinc oxide rotary kiln always functions in an optimal state. Our experiments show the strength and reliability of the proposed method, which reduces the usage of coal while maintaining high profits.

Surface oxides characterization based on hyperspectral observations

During steel sheets manufacturing, the addition of alloying elements to the steel to improve its mechanical performance can cause the formation of layers of selective oxides at the surface. These layers can significantly impact the surface properties of the steel sheets, which makes their detection an essential task. In this article, we present a novel methodology based on hyperspectral measurements to detect the presence of surface layers of selective oxides on steel sheets. Hyperspectral sensors have the ability to measure the reflectance of the steel surface over a large spectral band of the electromagnetic spectrum, usually in the infrared range. However, due to their limited spatial resolution, the observed reflectance spectra usually mix the contribution of several elementary oxides. A first step toward surface characterization is therefore to identify the elementary oxides contributing to the hyperspectral observations, a task referred to as hyperspectral unmixing in the literature. In our case, a major difficulty is that the optical model describing the formation of the electromagnetic wave coming onto the sensors is strongly nonlinear. To alleviate this difficulty, we rely on an approach that formulates the hyperspectral unmixing task as a sparse regression problem. This approach works by pre-computing the optical reflectance associated to reference layers of heterogeneous oxides that can form at the surface to construct a dictionary of reflectance spectra and by seeking to recover the hyperspectral observations as a linear combination of a small number of elements taken from this dictionary. To perform the recovery, we propose and compare two approaches based on the path and group Lasso algorithms. We illustrate our methodology on numerical experiments, which demonstrate that our approach is able to estimate the presence of selective oxides with high recall and precision. These numerical experiments also highlight the main limitations of the proposed approach. We finally apply our estimation method to characterize the surface chemistry of experimental samples. Our characterization approach is then compared to observations conducted on the samples through transmission electron microscopy and XPS analysis to demonstrate its validity.

Convex and Nonconvex Risk-Based Linear Regression at Scale

The value at risk (VaR) and the conditional value at risk (CVaR) are two popular risk measures to hedge against the uncertainty of data. In this paper, we provide a computational toolbox for solving high-dimensional sparse linear regression problems under either VaR or CVaR measures, the former being nonconvex and the latter convex. Unlike the empirical risk (neutral) minimization models in which the overall losses are decomposable across data, the aforementioned risk-sensitive models have nonseparable objective functions so that typical first order algorithms are not easy to scale. We address this scaling issue by adopting a semismooth Newton-based proximal augmented Lagrangian method of the convex CVaR linear regression problem. The matrix structures of the Newton systems are carefully explored to reduce the computational cost per iteration. The method is further embedded in a majorization–minimization algorithm as a subroutine to tackle the nonconvex VaR-based regression problem. We also discuss an adaptive sieving strategy to iteratively guess and adjust the effective problem dimension, which is particularly useful when a solution path associated with a sequence of tuning parameters is needed. Extensive numerical experiments on both synthetic and real data demonstrate the effectiveness of our proposed methods. In particular, they are about 53 times faster than the commercial package Gurobi for the CVaR-based sparse linear regression with 4,265,669 features and 16,087 observations. History: Accepted by Antonio Frangioni, Area Editor for Design & Analysis of Algorithms–Continuous. Funding: This work was supported in part by the NSF, the Division of Computing and Communication Foundations [Grant 2153352], the National Natural Science Foundation of China [Grant 12271187], and the Hong Kong Research Grant Council [Grant 15304019]. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2023.1282 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2022.0012 ) at ( http://dx.doi.org/10.5281/zenodo.7483279 ).

Neural network and Sparse identification of Nonlinear Dynamics Integrated Algorithm for Digital Twin identification

Digital twins play a critical role in simulating industrial manufacturing systems to increase productivity and reduce time spent on troubleshooting. Owing to the complexity of real-world industrial systems, automatic sparse identification has emerged as an attractive approach to perform digital twin modelling. The sparse identification of nonlinear dynamics (SINDy) is a machine learning algorithm that performs feature engineering by generating a model term library and then solves a sparse regression problem between the objective outputs and the generated features. By solving a linear-in-parameter sparse regression problem, SINDy provides automatic discovery of system governing equations. However, the performance of SINDy-based algorithms may decline dramatically when applied to identify complex nonlinear relationships, such as implicit relationships. The substantial number of input variables for a real industrial process may further complicate the modelling procedure. We therefore propose the neural network and SINDy integrated algorithm to automatically select the critical features from a model term library and utilize the neural network to capture the process nonlinearity that cannot be captured by a linear-in-parameter model. SINDy performs feature generation considering both numerical methods and first-principles knowledge, making the proposed algorithm a hybrid system identification approach. A diesel hydrotreating unit case study with 37 input variables is analyzed in this paper to demonstrate the advantages of the proposed algorithm for nonlinear digital twin identification. By combining the advantages from both SINDy and neural networks, the proposed algorithm is able to improve the output prediction accuracy for all the three objectives.

Detecting physical laws from data of stochastic dynamical systems perturbed by non-Gaussian α-stable Lévy noise

Massive data from observations, experiments and simulations of dynamical models in scientific and engineering fields make it desirable for data-driven methods to extract basic laws of these models. We present a novel method to identify such high dimensional stochastic dynamical systems that are perturbed by a non-Gaussian α-stable Lévy noise. More explicitly, firstly a machine learning framework to solve the sparse regression problem is established to grasp the drift terms through one of nonlocal Kramers–Moyal formulas. Then the jump measure and intensity of the noise are disposed by the relationship with statistical characteristics of the process. Three examples are then given to demonstrate the feasibility. This approach proposes an effective way to understand the complex phenomena of systems under non-Gaussian fluctuations and illuminates some insights into the exploration for further typical dynamical indicators such as the maximum likelihood transition path or mean exit time of these stochastic systems.

Permanent-Magnet Optimization for Stellarators as Sparse Regression

A common scientific inverse problem is the placement of magnets that produce a desired magnetic field inside a prescribed volume. This is a key component of stellarator design, and recently permanent magnets have been proposed as a potentially useful tool for magnetic field shaping. Here, we take a closer look at possible objective functions for permanent magnet optimization, reformulate the problem as sparse regression, and propose an algorithm that can efficiently solve many convex and nonconvex variants. The algorithm generates sparse solutions that are independent of the initial guess, explicitly enforces maximum strengths for the permanent magnets, and accurately produces the desired magnetic field. The algorithm is flexible, and our implementation is open-source and computationally fast. We conclude with two new permanent magnet configurations for the NCSX and MUSE stellarators. Our methodology can be additionally applied for effectively solving permanent magnet optimizations in other scientific fields, as well as for solving quite general high-dimensional, constrained, sparse regression problems, even if a binary solution is required.

Convex optimization under combinatorial sparsity constraints

We present a heuristic approach for convex optimization problems containing different types of sparsity constraints. Whenever the support is required to belong to a matroid, we propose an exchange heuristic adapting the support in every iteration. The entering non-zero is determined by considering the dual multipliers of the bounds on variables being fixed to zero. While this algorithm is purely heuristic, we show experimentally that it often finds near-optimal solutions for cardinality-constrained knapsack problems and for sparse regression problems.

LARGE-SCALE MULTIVARIATE SPARSE REGRESSION WITH APPLICATIONS TO UK BIOBANK.

In high-dimensional regression problems, often a relatively small subset of the features are relevant for predicting the outcome, and methods that impose sparsity on the solution are popular. When multiple correlated outcomes are available (multitask), reduced rank regression is an effective way to borrow strength and capture latent structures that underlie the data. Our proposal is motivated by the UK Biobank population-based cohort study, where we are faced with large-scale, ultrahigh-dimensional features, and have access to a large number of outcomes (phenotypes)-lifestyle measures, biomarkers, and disease outcomes. We are hence led to fit sparse reduced-rank regression models, using computational strategies that allow us to scale to problems of this size. We use a scheme that alternates between solving the sparse regression problem and solving the reduced rank decomposition. For the sparse regression component we propose a scalable iterative algorithm based on adaptive screening that leverages the sparsity assumption and enables us to focus on solving much smaller subproblems. The full solution is reconstructed and tested via an optimality condition to make sure it is a valid solution for the original problem. We further extend the method to cope with practical issues, such as the inclusion of confounding variables and imputation of missing values among the phenotypes. Experiments on both synthetic data and the UK Biobank data demonstrate the effectiveness of the method and the algorithm. We present multiSnpnet package, available at http://github.com/junyangq/multiSnpnet that works on top of PLINK2 files, which we anticipate to be a valuable tool for generating polygenic risk scores from human genetic studies.

SINDy-SA framework: enhancing nonlinear system identification with sensitivity analysis.

Machine learning methods have revolutionized studies in several areas of knowledge, helping to understand and extract information from experimental data. Recently, these data-driven methods have also been used to discover structures of mathematical models. The sparse identification of nonlinear dynamics (SINDy) method has been proposed with the aim of identifying nonlinear dynamical systems, assuming that the equations have only a few important terms that govern the dynamics. By defining a library of possible terms, the SINDy approach solves a sparse regression problem by eliminating terms whose coefficients are smaller than a threshold. However, the choice of this threshold is decisive for the correct identification of the model structure. In this work, we build on the SINDy method by integrating it with a global sensitivity analysis (SA) technique that allows to hierarchize terms according to their importance in relation to the desired quantity of interest, thus circumventing the need to define the SINDy threshold. The proposed SINDy-SA framework also includes the formulation of different experimental settings, recalibration of each identified model, and the use of model selection techniques to select the best and most parsimonious model. We investigate the use of the proposed SINDy-SA framework in a variety of applications. We also compare the results against the original SINDy method. The results demonstrate that the SINDy-SA framework is a promising methodology to accurately identify interpretable data-driven models.Supplementary InformationThe online version contains supplementary material available at 10.1007/s11071-022-07755-2.

The Price of Competition: Effect Size Heterogeneity Matters in High Dimensions

In high-dimensional sparse regression, would increasing the signal-to-noise ratio while fixing the sparsity level always lead to better model selection? For high-dimensional sparse regression problems, surprisingly, in this paper we answer this question in the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">negative</i> in the regime of linear sparsity for the Lasso method, relying on a new concept we term <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">effect size heterogeneity</i> . Roughly speaking, a regression coefficient vector has high effect size heterogeneity if its nonzero entries have significantly different magnitudes. From the viewpoint of this new measure, we prove that the false and true positive rates achieve the optimal trade-off <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">uniformly</i> along the Lasso path when this measure is maximal in a certain sense, and the worst trade-off is achieved when it is minimal in the sense that all nonzero effect sizes are roughly equal. Moreover, we demonstrate that the first false selection occurs much earlier when effect size heterogeneity is minimal than when it is maximal. The underlying cause of these two phenomena is, metaphorically speaking, the “competition” among variables with effect sizes of the same magnitude in entering the model. Taken together, our findings suggest that effect size heterogeneity shall serve as an important complementary measure to the sparsity of regression coefficients in the analysis of high-dimensional regression problems. Our proofs use techniques from approximate message passing theory as well as a novel technique for estimating the rank of the first false variable.

A Geometric Proximal Gradient Method for Sparse Least Squares Regression with Probabilistic Simplex Constraint

In this paper, we consider the sparse least squares regression problem with probabilistic simplex constraint. Due to the probabilistic simplex constraint, one could not apply the L1 regularization to the considered regression model. To find a sparse solution, we reformulate the least squares regression problem as a nonconvex and nonsmooth L1 regularized minimization problem over the unit sphere. Then we propose a geometric proximal gradient method for solving the regularized problem, where the explicit expression of the global solution to every involved subproblem is obtained. The global convergence of the proposed method is established under some mild assumptions. Some numerical results are reported to illustrate the effectiveness of the proposed algorithm.