The application of regularisation to variable selection in statistical modelling

Abstract

The aim of variable selection is the identification of the most important predictors that define the response of a linear system. Many techniques for variable selection use a constrained least squares (LS) formulation in which the constraint is imposed in the 1-norm (the lasso), or the 2-norm (Tikhonov regularisation), or a linear combination of these norms (the elastic net). It is always assumed that a constraint must necessarily be imposed, but the consequences of its imposition have not been addressed. This assumption is considered in this paper and it is shown that the correct application of Tikhonov regularisation to the LS problem min‖Ax−b‖2 requires that A and b satisfy a condition C. If this condition is satisfied, then the solution of the LS problem with this constraint is numerically stable and the regularisation error e between the solution of this problem and the solution of the LS problem is small. If, however, the condition C is not satisfied, then the error e is large. The condition C is derived from a refined normwise condition number of the solution of the LS problem. The paper includes examples of regularisation and variable selection with correlated variables that illustrate the theory.