High-dimensional Sparse Linear Regression Research Articles

Multiple-hypothesis testing rules for high-dimensional model selection and sparse-parameter estimation

We consider the problem of model selection for high-dimensional sparse linear regression models. We pose the model selection problem as a multiple-hypothesis testing problem and employ the methods of false discovery rate (FDR) and familywise error rate (FER) to solve it. We also present the reformulation of the FDR/FER-based approaches as criterion-based model selection rules and establish their relation to the extended Bayesian Information Criterion (EBIC), which is a state-of-the-art high-dimensional model selection rule. We use numerical simulations to show that the proposed FDR/FER method is well suited for high-dimensional model selection and performs better than EBIC.

Convex and Nonconvex Risk-Based Linear Regression at Scale

The value at risk (VaR) and the conditional value at risk (CVaR) are two popular risk measures to hedge against the uncertainty of data. In this paper, we provide a computational toolbox for solving high-dimensional sparse linear regression problems under either VaR or CVaR measures, the former being nonconvex and the latter convex. Unlike the empirical risk (neutral) minimization models in which the overall losses are decomposable across data, the aforementioned risk-sensitive models have nonseparable objective functions so that typical first order algorithms are not easy to scale. We address this scaling issue by adopting a semismooth Newton-based proximal augmented Lagrangian method of the convex CVaR linear regression problem. The matrix structures of the Newton systems are carefully explored to reduce the computational cost per iteration. The method is further embedded in a majorization–minimization algorithm as a subroutine to tackle the nonconvex VaR-based regression problem. We also discuss an adaptive sieving strategy to iteratively guess and adjust the effective problem dimension, which is particularly useful when a solution path associated with a sequence of tuning parameters is needed. Extensive numerical experiments on both synthetic and real data demonstrate the effectiveness of our proposed methods. In particular, they are about 53 times faster than the commercial package Gurobi for the CVaR-based sparse linear regression with 4,265,669 features and 16,087 observations. History: Accepted by Antonio Frangioni, Area Editor for Design & Analysis of Algorithms–Continuous. Funding: This work was supported in part by the NSF, the Division of Computing and Communication Foundations [Grant 2153352], the National Natural Science Foundation of China [Grant 12271187], and the Hong Kong Research Grant Council [Grant 15304019]. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2023.1282 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2022.0012 ) at ( http://dx.doi.org/10.5281/zenodo.7483279 ).

A communication-efficient method for ℓ0 regularization linear regression models

We propose a communication-efficient distributed learning algorithm for high-dimensional sparse linear regression models in the scenario that the data are stored across multiple machines. Our approach is a distributed version of the SDAR [Huang J, Jiao Y, Liu Y, et al. A constructive approach to l0 penalized regression. J Mach Learn Res. 2018;19(1):403–439] method for solving the KKT system of the regularized least squares. At each step of the proposed method, the reduced least squares are solved by the steepest descent method, which only needs to calculate the gradient vectors on each node machine and communicate them instead of the data. We refer to this as SD-SDAR for brevity. Under some regular conditions, we obtain the sharp and error bounds for the solution sequences generated by SD-SDAR algorithm. We investigate the computational complexity and show that the number of rounds of communications are bounded by and , respectively, where J is the number of important predictors, R is the relative magnitude of the non-zero target coefficients, N is the total sample size and p is the dimension of covariates. Simulation studies illustrate that SD-SDAR outperforms some existing distributed methods in accuracy, efficiency and support recovery.

A kind of new time-weighted nonnegative lasso index-tracking model and its application

This study describes improved index-tracking methods to replicate the target index’s market performance in a high-dimensional sparse linear regression with nonnegative constraints on the coefficients. The main objective of this study is to construct a sparse portfolio with a better prediction effect and robustness. Considering the influence of time factors on index tracking, we propose a time-weighted nonnegative lasso index tracking model under different market constraints and define two new time-weighted construction methods. This index tracking model is an extension of Lasso and has variable selection consistency and estimation consistency under time-weighted nonnegative irrepresentable conditions similar to the irrepresentable condition in Lasso. We use the multiplicative updates algorithm to obtain the model’s solution since it is faster and simpler. The constrained index tracking problem in the stock market without short sales is studied in the latter part. The empirical results indicate that the optimized time-weighted nonnegative lasso index tracking model can obtain a smaller out-of-sample tracking error. The constructed portfolio has a better prediction effect and robustness, and we find that the exponential time-weighted method is better than the linear time-weighted method in capturing time information.

Does SLOPE outperform bridge regression?

Abstract A recently proposed SLOPE estimator [6] has been shown to adaptively achieve the minimax $\ell _2$ estimation rate under high-dimensional sparse linear regression models [25]. Such minimax optimality holds in the regime where the sparsity level $k$, sample size $n$ and dimension $p$ satisfy $k/p\rightarrow 0, k\log p/n\rightarrow 0$. In this paper, we characterize the estimation error of SLOPE under the complementary regime where both $k$ and $n$ scale linearly with $p$, and provide new insights into the performance of SLOPE estimators. We first derive a concentration inequality for the finite sample mean square error (MSE) of SLOPE. The quantity that MSE concentrates around takes a complicated and implicit form. With delicate analysis of the quantity, we prove that among all SLOPE estimators, LASSO is optimal for estimating $k$-sparse parameter vectors that do not have tied nonzero components in the low noise scenario. On the other hand, in the large noise scenario, the family of SLOPE estimators are sub-optimal compared with bridge regression such as the Ridge estimator.

Hard thresholding regression.

In this paper, we propose the hard thresholding regression (HTR) for estimating high-dimensional sparse linear regression models. HTR uses a two-stage convex algorithm to approximate the ℓ 0-penalized regression: The first stage calculates a coarse initial estimator, and the second stage identifies the oracle estimator by borrowing information from the first one. Theoretically, the HTR estimator achieves the strong oracle property over a wide range of regularization parameters. Numerical examples and a real data example lend further support to our proposed methodology.

Optimal estimation of slope vector in high-dimensional linear transformation models

In a linear transformation model, there exists an unknown monotone, typically nonlinear, transformation function such that the transformed response variable is related to the predictor variables via a linear regression model. This paper presents CENet, a new method for estimating the slope vector and simultaneously performing variable selection in the high-dimensional sparse linear transformation model. CENet is the solution to a convex optimization problem which can be computed efficiently from an algorithm with guaranteed convergence to the global optimum. It is shown that when the joint distribution of the predictors and errors is elliptical, under some regularity conditions, CENet attains the same optimal rate of convergence as the best regression method in the high-dimensional sparse linear regression model. The empirical performance of CENet is shown on both simulated and real datasets. The connection of CENet with existing nonlinear regression/multivariate methods is also discussed.

Optimal prediction for sparse linear models? Lower bounds for coordinate-separable M-estimators

For the problem of high-dimensional sparse linear regression, it is known that an $\ell_{0}$-based estimator can achieve a $1/n$ “fast” rate for prediction error without any conditions on the design matrix, whereas in the absence of restrictive conditions on the design matrix, popular polynomial-time methods only guarantee the $1/\sqrt{n}$ “slow” rate. In this paper, we show that the slow rate is intrinsic to a broad class of M-estimators. In particular, for estimators based on minimizing a least-squares cost function together with a (possibly nonconvex) coordinate-wise separable regularizer, there is always a “bad” local optimum such that the associated prediction error is lower bounded by a constant multiple of $1/\sqrt{n}$. For convex regularizers, this lower bound applies to all global optima. The theory is applicable to many popular estimators, including convex $\ell_{1}$-based methods as well as M-estimators based on nonconvex regularizers, including the SCAD penalty or the MCP regularizer. In addition, we show that bad local optima are very common, in that a broad class of local minimization algorithms with random initialization typically converge to a bad solution.

Nonnegative-lasso and application in index tracking

This paper proposes the nonnegative-lasso method for variable selection in high dimensional sparse linear regression models with the nonnegative constraints on the coefficients. This method is an extension of Lasso and is shown to have variable selection consistency and estimation consistency under certain condition similar to Irrepresentable Condition in Lasso. To get the solution of the nonnegative-lasso, many algorithms such as Lars, coordinate decent can be used, among which multiplicative updates approach is preferred since it is faster and simpler. The constrained index tracking problem in stock market without short sales is studied in the latter part. The tracking results indicate that nonnegative-lasso can get small tracking error and is successful in assets selection.

The [formula omitted] penalized LAD estimator for high dimensional linear regression

In this paper, the high-dimensional sparse linear regression model is considered, where the overall number of variables is larger than the number of observations. We investigate the L1 penalized least absolute deviation method. Different from most of the other methods, the L1 penalized LAD method does not need any knowledge of standard deviation of the noises or any moment assumptions of the noises. Our analysis shows that the method achieves near oracle performance, i.e. with large probability, the L2 norm of the estimation error is of order O(klogp/n). The result is true for a wide range of noise distributions, even for the Cauchy distribution. Numerical results are also presented.

The adaptive lasso in high-dimensional sparse heteroscedastic models

In this paper we study the asymptotic properties of the adaptive Lasso estimate in high-dimensional sparse linear regression models with heteroscedastic errors. It is demonstrated that model selection properties and asymptotic normality of the selected parameters remain valid but with a suboptimal asymptotic variance. A weighted adaptive Lasso estimate is introduced and investigated. In particular, it is shown that the new estimate performs consistent model selection and that linear combinations of the estimates corresponding to the non-vanishing components are asymptotically normally distributed with a smaller variance than those obtained by the “classical” adaptive Lasso. The results are illustrated in a data example and by means of a small simulation study.

High-dimensional regression with noisy and missing data: Provable guarantees with nonconvexity

Although the standard formulations of prediction problems involve fully-observed and noiseless data drawn in an i.i.d. manner, many applications involve noisy and/or missing data, possibly involving dependence, as well. We study these issues in the context of high-dimensional sparse linear regression, and propose novel estimators for the cases of noisy, missing and/or dependent data. Many standard approaches to noisy or missing data, such as those using the EM algorithm, lead to optimization problems that are inherently nonconvex, and it is difficult to establish theoretical guarantees on practical algorithms. While our approach also involves optimizing nonconvex programs, we are able to both analyze the statistical error associated with any global optimum, and more surprisingly, to prove that a simple algorithm based on projected gradient descent will converge in polynomial time to a small neighborhood of the set of all global minimizers. On the statistical side, we provide nonasymptotic bounds that hold with high probability for the cases of noisy, missing and/or dependent data. On the computational side, we prove that under the same types of conditions required for statistical consistency, the projected gradient descent algorithm is guaranteed to converge at a geometric rate to a near-global minimizer. We illustrate these theoretical predictions with simulations, showing close agreement with the predicted scalings.

Square-root lasso: pivotal recovery of sparse signals via conic programming

We propose a pivotal method for estimating high-dimensional sparse linear regression models, where the overall number of regressors $p$ is large, possibly much larger than $n$, but only $s$ regressors are significant. The method is a modification of the lasso, called the square-root lasso. The method is pivotal in that it neither relies on the knowledge of the standard deviation $\sigma$ or nor does it need to pre-estimate $\sigma$. Moreover, the method does not rely on normality or sub-Gaussianity of noise. It achieves near-oracle performance, attaining the convergence rate $\sigma \{(s/n)\log p\}^{1/2}$ in the prediction norm, and thus matching the performance of the lasso with known $\sigma$. These performance results are valid for both Gaussian and non-Gaussian errors, under some mild moment restrictions. We formulate the square-root lasso as a solution to a convex conic programming problem, which allows us to implement the estimator using efficient algorithmic methods, such as interior-point and first-order methods.

Square-Root Lasso: Pivotal Recovery of Sparse Signals via Conic Programming

We propose a pivotal method for estimating high-dimensional sparse linear regression models, where the overall number of regressors p is large, possibly much larger than n, but only s regressors are significant. The method is a modification of the lasso, called the square-root lasso. The method is pivotal in that it neither relies on the knowledge of the standard deviation σ or nor does it need to pre-estimate σ. Moreover, the method does not rely on normality or sub-Gaussianity of noise. It achieves near-oracle performance, attaining the convergence rate σ{(s/n) log p}1/2 in the prediction norm, and thus matching the performance of the lasso with known σ. These performance results are valid for both Gaussian and non-Gaussian errors, under some mild moment restrictions. We formulate the square-root lasso as a solution to a convex conic programming problem, which allows us to implement the estimator using efficient algorithmic methods, such as interior-point and first-order methods.

Consistent tuning parameter selection in high dimensional sparse linear regression

An exhaustive search as required for traditional variable selection methods is impractical in high dimensional statistical modeling. Thus, to conduct variable selection, various forms of penalized estimators with good statistical and computational properties, have been proposed during the past two decades. The attractive properties of these shrinkage and selection estimators, however, depend critically on the size of regularization which controls model complexity. In this paper, we consider the problem of consistent tuning parameter selection in high dimensional sparse linear regression where the dimension of the predictor vector is larger than the size of the sample. First, we propose a family of high dimensional Bayesian Information Criteria (HBIC), and then investigate the selection consistency, extending the results of the extended Bayesian Information Criterion (EBIC), in Chen and Chen (2008) to ultra-high dimensional situations. Second, we develop a two-step procedure, the SIS+AENET, to conduct variable selection in p > n situations. The consistency of tuning parameter selection is established under fairly mild technical conditions. Simulation studies are presented to confirm theoretical findings, and an empirical example is given to illustrate the use in the internet advertising data.

FIRST: Combining forward iterative selection and shrinkage in high dimensional sparse linear regression

We propose a new class of variable selection techniques for regression in high dimensional linear models based on a forward selection version of the LASSO, adaptive LASSO or elastic net, respectively to be called as forward iterative regression and shrinkage technique (FIRST), adaptive FIRST and elastic FIRST. These methods seem to work effectively for extremely sparse high dimensional linear models. We exploit the fact that the LASSO, adaptive LASSO and elastic net have closed form solutions when the predictor is onedimensional. The explicit formula is then repeatedly used in an iterative fashion to build the model until convergence occurs. By carefully considering the relationship between estimators at successive stages, we develop fast algorithms to compute our estimators. The performance of our new estimators are compared with commonly used estimators in terms of predictive accuracy and errors in variable selection. AMS 2000 subject classifications: Primary 62J05, 62J05; secondary 62J07.