Square-root Lasso Research Articles

An Enhanced Extreme Learning Machine Based on Square-Root Lasso Method

Extreme learning machine (ELM) is one of the most notable machine learning algorithms with many advantages, especially its training speed. However, ELM has some drawbacks such as instability, poor generalizability and overfitting in the case of multicollinearity in the linear model. This paper introduces square-root lasso ELM (SQRTL-ELM) as a novel regularized ELM algorithm to deal with these drawbacks of ELM. A modified version of the alternating minimization algorithm is used to obtain the estimates of the proposed method. Various techniques are presented to determine the tuning parameter of SQRTL-ELM. The method is compared with the basic ELM, RIDGE-ELM, LASSO-ELM and ENET-ELM on six benchmark data sets. Performance evaluation results show that the SQRTL-ELM exhibits satisfactory performance in terms of testing root mean squared error in benchmark data sets for the sake of slightly extra computation time. The superiority level of the method depends on the tuning parameter selection technique. As a result, the proposed method can be considered a powerful alternative to avoid performance loss in regression problems .

Square-root lasso under correlated regressors: Tight statistical analysis with a wireless communications application

<p>This paper provided a comprehensive analysis of sparse signal estimation from noisy and possibly underdetermined linear observations in the high-dimensional asymptotic regime. The focus was on the square-root lasso (sqrt-lasso), a popular convex optimization method used for sparse signal recovery. We analyzed its performance using several metrics, such as root-mean-squared error (r.m.s.e.), mean absolute error (m.a.e.), cosine similarity, and true/false recovery rates. The analysis assumed a normally distributed design matrix with left-sided correlation and Gaussian noise. In addition to theoretical contributions, we applied these results to a real-world wireless communications problem by examining the error performance of sqrt-lasso in generalized space shift keying (GSSK) modulation for multiple-input multiple-output (MIMO) systems. This application was particularly relevant, as the GSSK modulation generates sparse data symbols, making it an ideal scenario for sparse recovery techniques. Our study offered tight asymptotic approximations for the performance of sqrt-lasso in such systems. Beyond the wireless communications application, the results had broader implications for other high-dimensional applications, including compressed sensing, machine learning, and statistical inference. The analysis presented in this paper, supported by numerical simulations, provided practical insights into how sqrt-lasso behaved under correlated designs, offering useful guidelines for optimizing its use in real-world scenarios. The expressions and insights obtained from this study can be used to optimally choose the penalization parameter of the sqrt-lasso. By applying these results, one can make informed decisions about performance and fine-tuning the sqrt-lasso, considering the presence of correlated regressors in a high-dimensional context.</p>

Optimal Tuning-Free Convex Relaxation for Noisy Matrix Completion

This paper is concerned with noisy matrix completion—the problem of recovering a low-rank matrix from partial and noisy entries. Under uniform sampling and incoherence assumptions, we prove that a tuning-free square-root matrix completion estimator (square-root MC) achieves optimal statistical performance for solving the noisy matrix completion problem. Similar to the square-root Lasso estimator in high-dimensional linear regression, square-root MC does not rely on the knowledge of the size of the noise. While solving square-root MC is a convex program, our statistical analysis of square-root MC hinges on its intimate connections to a nonconvex rank-constrained estimator.

Embracing off-the-grid samples

Many empirical studies suggest that samples of continuous-time signals taken at locations randomly deviated from an equispaced grid (i.e., off-the-grid) can benefit signal acquisition, e.g., undersampling and anti-aliasing. However, explicit statements of such advantages and their respective conditions are scarce in the literature. This paper provides some insight on this topic when the sampling positions are known, with grid deviations generated i.i.d. from a variety distributions. By solving a square-root LASSO decoder with an interpolation kernel we demonstrate the capabilities of nonuniform samples for compressive sampling, an effective paradigm for undersampling and anti-aliasing. For functions in the Wiener algebra that admit a discrete s-sparse representation in some transform domain, we show that O(spolylogN)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${\\mathcal {O}}(s{{\\,\\mathrm{poly\\,\\hspace{-2pt}log}\\,}}N)$$\\end{document} random off-the-grid samples are sufficient to recover an accurate N2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\frac{N}{2}$$\\end{document}-bandlimited approximation of the signal. For sparse signals (i.e., s≪N\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$s \\ll N$$\\end{document}), this sampling complexity is a great reduction in comparison to equispaced sampling where O(N)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${\\mathcal {O}}(N)$$\\end{document} measurements are needed for the same quality of reconstruction (Nyquist–Shannon sampling theorem). We further consider noise attenuation via oversampling (relative to a desired bandwidth), a standard technique with limited theoretical understanding when the sampling positions are non-equispaced. By solving a least squares problem, we show that O(NlogN)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${\\mathcal {O}}(N\\log N)$$\\end{document} i.i.d. randomly deviated samples provide an accurate N2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\frac{N}{2}$$\\end{document}-bandlimited approximation of the signal with suppression of the noise energy by a factor ∼1log(N).\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\sim \\frac{1}{\\sqrt{\\log (N)}}.$$\\end{document}

A Dual Semismooth Newton Based Augmented Lagrangian Method for Large-Scale Linearly Constrained Sparse Group Square-Root Lasso Problems

A Dual Semismooth Newton Based Augmented Lagrangian Method for Large-Scale Linearly Constrained Sparse Group Square-Root Lasso Problems

Smooth over-parameterized solvers for non-smooth structured optimization

Non-smooth optimization is a core ingredient of many imaging or machine learning pipelines. Non-smoothness encodes structural constraints on the solutions, such as sparsity, group sparsity, low-rank and sharp edges. It is also the basis for the definition of robust loss functions and scale-free functionals such as square-root Lasso. Standard approaches to deal with non-smoothness leverage either proximal splitting or coordinate descent. These approaches are effective but usually require parameter tuning, preconditioning or some sort of support pruning. In this work, we advocate and study a different route, which operates a non-convex but smooth over-parameterization of the underlying non-smooth optimization problems. This generalizes quadratic variational forms that are at the heart of the popular Iterative Reweighted Least Squares. Our main theoretical contribution connects gradient descent on this reformulation to a mirror descent flow with a varying Hessian metric. This analysis is crucial to derive convergence bounds that are dimension-free. This explains the efficiency of the method when using small grid sizes in imaging. Our main algorithmic contribution is to apply the Variable Projection method which defines a new formulation by explicitly minimizing over part of the variables. This leads to a better conditioning of the minimized functional and improves the convergence of simple but very efficient gradient-based methods, for instance quasi-Newton solvers. We exemplify the use of this new solver for the resolution of regularized regression problems for inverse problems and supervised learning, including total variation prior and non-convex regularizers.

The sparsity of LASSO-type minimizers

This note extends an attribute of the LASSO procedure to a whole class of related procedures, including square-root LASSO, square LASSO, LAD-LASSO, and an instance of generalized LASSO. Namely, under the assumption that the input matrix satisfies an ℓ p -restricted isometry property (which in some sense is weaker than the standard ℓ 2 -restricted isometry property assumption), it is shown that if the input vector comes from the exact measurement of a sparse vector, then the minimizer of any such LASSO-type procedure has sparsity comparable to the sparsity of the measured vector. The result remains valid in the presence of moderate measurement error when the regularization parameter is not too small.

WARPd: A Linearly Convergent First-Order Primal-Dual Algorithm for Inverse Problems with Approximate Sharpness Conditions

Sharpness conditions directly control the recovery performance of restart schemes for first-order optimization methods without the need for restrictive assumptions such as strong convexity. However, they are challenging to apply in the presence of noise or approximate model classes (e.g., approximate sparsity). We provide a first-order method: weighted, accelerated, and restarted primal-dual (WARPd), based on primal-dual iterations and a novel restart-reweight scheme. Under a generic approximate sharpness condition, WARPd achieves stable linear convergence to the desired vector. Many problems of interest fit into this framework. For example, we analyze sparse recovery in compressed sensing, low-rank matrix recovery, matrix completion, TV regularization, minimization of $\|Bx\|_{l^1}$ under constraints ($l^1$-analysis problems for general $B$), and mixed regularization problems. We show how several quantities controlling recovery performance also provide explicit approximate sharpness constants. Numerical experiments show that WARPd compares favorably with specialized state-of-the-art methods and is ideally suited for solving large-scale problems. We also present a noise-blind variant based on a square-root LASSO decoder. Finally, we show how to unroll WARPd as neural networks. This approximation theory result provides lower bounds for stable and accurate neural networks for inverse problems and sheds light on architecture choices. Code and a gallery of examples are available online as a MATLAB package.

Uniform inference in high-dimensional Gaussian graphical models

Summary Graphical models have become a popular tool for representing dependencies within large sets of variables and are crucial for representing causal structures. We provide results for uniform inference on high-dimensional graphical models, in which the number of target parameters $d$ is potentially much larger than the sample size, under approximate sparsity. Our results highlight how graphical models can be estimated and recovered using modern machine learning methods in high-dimensional complex settings. To construct simultaneous confidence regions on many target parameters, it is crucial to have sufficiently fast estimation rates of the nuisance functions. In this context, we establish uniform estimation rates and sparsity guarantees for the square-root lasso estimator in a random design under approximate sparsity conditions. These might be of independent interest for related problems in high dimensions. We also demonstrate in a comprehensive simulation study that our procedure has good small sample properties in comparison to existing methods, and we present two empirical applications.

Thresholding tests based on affine LASSO to achieve non-asymptotic nominal level and high power under sparse and dense alternatives in high dimension

Thresholding estimators such as the existing square-root and LAD LASSO, and the new affine and GLM LASSO with new link functions, have the ability to set coefficients to zero. They will yield new pivotal statistics which enjoy high power under sparse or dense alternative hypotheses. Under a general formalism, thresholding tests not only recover existing tests such as Rao score test and Fisher nonparametric sign test, but also unveil new tests, for the global/omnibus hypothesis in high dimension in particular. Although pivotal, the new statistics do not have a known distribution, so the critical value of the test is calculated by straightforward Monte Carlo, which yields exact level and high power as illustrated on simulated data.

High-dimensional inference robust to outliers with -norm penalization

This article studies inference in the high-dimensional linear regression model with outliers. Sparsity constraints are imposed on the vector of coefficients of the covariates. The number of outliers can grow with the sample size while their proportion goes to 0. We propose a two-step procedure for inference on the coefficients of a fixed subset of regressors. The first step is a based on several square-root lasso -norm penalized estimators, while the second step is the ordinary least squares estimator applied to a well-chosen regression. We establish asymptotic normality of the two-step estimator. The proposed procedure is efficient in the sense that it attains the semiparametric efficiency bound when applied to the model without outliers under homoscedasticity. This approach is also computationally advantageous, it amounts to solving a finite number of convex optimization programs.

Sensitivity analysis of Wasserstein distributionally robust optimization problems.

We consider sensitivity of a generic stochastic optimization problem to model uncertainty. We take a non-parametric approach and capture model uncertainty using Wasserstein balls around the postulated model. We provide explicit formulae for the first-order correction to both the value function and the optimizer and further extend our results to optimization under linear constraints. We present applications to statistics, machine learning, mathematical finance and uncertainty quantification. In particular, we provide an explicit first-order approximation for square-root LASSO regression coefficients and deduce coefficient shrinkage compared to the ordinary least-squares regression. We consider robustness of call option pricing and deduce a new Black–Scholes sensitivity, a non-parametric version of the so-called Vega. We also compute sensitivities of optimized certainty equivalents in finance and propose measures to quantify robustness of neural networks to adversarial examples.

A fast algorithm for group square-root Lasso based group-sparse regression

Group square-root Lasso (GSRL) is a promising tool for group-sparse regression since the hyperparameter is independent of noise level. Recent works also reveal its connections to some statistically sound and hyperparameter-free methods, e.g., group-sparse iterative covariance-based estimation (GSPICE). However, the non-smoothness of the data-fitting term leads to the difficulty in solving the optimization problem of GSRL, and available solvers usually suffer either slow convergence or restrictions on the dictionary. In this paper, we propose a class of efficient solvers for GSRL in a block coordinate descent manner, including group-wise cyclic minimization (GCM) for group-wise orthonormal dictionary and generalized GCM (G-GCM) for general dictionary. Both strict descent property and global convergence are proved. To cope with signal processing applications, the complex-valued multiple measurement vectors (MMV) case is considered. The proposed algorithm can also be used for the fast implementation of methods with theoretical equivalence to GSRL, e.g., GSPICE. Significant superiority in computational efficiency is verified by simulation results.

Confidence regions in Wasserstein distributionally robust estimation

Summary Estimators based on Wasserstein distributionally robust optimization are obtained as solutions of min-max problems in which the statistician selects a parameter minimizing the worst-case loss among all probability models within a certain distance from the underlying empirical measure in a Wasserstein sense. While motivated by the need to identify optimal model parameters or decision choices that are robust to model misspecification, these distributionally robust estimators recover a wide range of regularized estimators, including square-root lasso and support vector machines, among others. This paper studies the asymptotic normality of these distributionally robust estimators as well as the properties of an optimal confidence region induced by the Wasserstein distributionally robust optimization formulation. In addition, key properties of min-max distributionally robust optimization problems are also studied; for example, we show that distributionally robust estimators regularize the loss based on its derivative, and we also derive general sufficient conditions which show the equivalence between the min-max distributionally robust optimization problem and the corresponding max-min formulation.

On the Performance of the SPICE Method

This letter considers the performance of the SPICE method in the context of line spectrum estimation by exploiting its equivalent square-root lasso (SR-LASSO) representation established in previous work. The inefficiencies of existing analyses are that the guarantees are not applicable to deterministic Fourier measurement matrices, and the studied range of the tuning parameter excludes the SR-LASSO formulation of the SPICE method. The key observation of this letter is that, for a particular range of tuning parameter, the solutions which overfit the noisy measurements are valid SR-LASSO solutions. Based on this observation, we propose to analyze these overfitting solutions and derive the condition of exact on-grid source localization that is applicable to the SPICE method. The numerical experiments demonstrate our theoretical claims.

广义线性模型的平方根Lasso选择性推断

广义线性模型的平方根Lasso选择性推断

Lassopack: Model selection and prediction with regularized regression in Stata

In this article, we introduce lassopack, a suite of programs for regularized regression in Stata. lassopack implements lasso, square-root lasso, elastic net, ridge regression, adaptive lasso, and postestimation ordinary least squares. The methods are suitable for the high-dimensional setting, where the number of predictors p may be large and possibly greater than the number of observations, n. We offer three approaches for selecting the penalization (“tuning”) parameters: information criteria (implemented in lasso2), K-fold cross-validation and h-step-ahead rolling cross-validation for cross-section, panel, and time-series data (cvlasso), and theory-driven (“rigorous” or plugin) penalization for the lasso and square-root lasso for cross-section and panel data (rlasso). We discuss the theoretical framework and practical considerations for each approach. We also present Monte Carlo results to compare the performances of the penalization approaches.

Variable selection for sparse logistic regression

We consider the variable selection problem in a sparse logistical regression model. Inspired by the square-root Lasso, we develop a weighted score Lasso for logistical regression. The new method yields the estimation $${\ell }_1$$ error bound under similar assumptions as introduced in Bach et al. (Electron J Stat 4:384–414, 2010). Compared to standard Lasso, the weighted score Lasso provides a direct choice for the tuning parameter. Both theoretical and simulation results confirm the satisfactory performance of the proposed method. We illustrate our methodology with a real microarray data set.

Inference robust to outliers with ℓ1-norm penalization

This paper considers inference in a linear regression model with outliers in which the number of outliers can grow with sample size while their proportion goes to 0. We propose a square-root lasso ℓ1-norm penalized estimator. We derive rates of convergence and establish asymptotic normality. Our estimator has the same asymptotic variance as the OLS estimator in the standard linear model. This enables us to build tests and confidence sets in the usual and simple manner. The proposed procedure is also computationally advantageous, it amounts to solving a convex optimization program. Overall, the suggested approach offers a practical robust alternative to the ordinary least squares estimator.

Correcting for unknown errors in sparse high-dimensional function approximation

We consider sparsity-based techniques for the approximation of high-dimensional functions from random pointwise evaluations. To date, almost all the works published in this field contain some a priori assumptions about the error corrupting the samples that are hard to verify in practice. In this paper, we instead focus on the scenario where the error is unknown. We study the performance of four sparsity-promoting optimization problems: weighted quadratically-constrained basis pursuit, weighted LASSO, weighted square-root LASSO, and weighted LAD-LASSO. From the theoretical perspective, we prove uniform recovery guarantees for these decoders, deriving recipes for the optimal choice of the respective tuning parameters. On the numerical side, we compare them in the pure function approximation case and in applications to uncertainty quantification of ODEs and PDEs with random inputs. Our main conclusion is that the lesser-known square-root LASSO is better suited for high-dimensional approximation than the other procedures in the case of bounded noise, since it avoids (both theoretically and numerically) the need for parameter tuning.