Numerical properties of solutions of LASSO regression

Abstract

The determination of a concise model of a linear system when there are fewer samples m than predictors n requires the solution of the equation Ax=b, where A∈Rm×n and rankA=m, such that the selected solution from the infinite number of solutions is sparse, that is, many of its components are zero. This leads to the minimisation with respect to x of f(x,λ)=‖Ax−b‖22+λ‖x‖1, where λ is the regularisation parameter. This problem, which is called LASSO regression, yields a family of functions xlasso(λ) and it is necessary to determine the optimal value of λ, that is, the value of λ that balances the fidelity of the model, ‖Axlasso(λ)−b‖≈0, and the satisfaction of the constraint that xlasso(λ) be sparse. The aim of this paper is an investigation of the numerical properties of xlasso(λ), and the main conclusion of this investigation is the incompatibility of sparsity and stability, that is, a sparse solution xlasso(λ) that preserves the fidelity of the model exists if the least squares (LS) solution xls=A†b is unstable. Two methods, cross validation and the L-curve, for the computation of the optimal value of λ are compared and it is shown that the L-curve yields significantly better results. This difference between stable and unstable solutions xls of the LS problem manifests itself in the very different forms of the L-curve for these two solutions. The paper includes examples of stable and unstable solutions xls that demonstrate the theory.