On the Best Choice of Lasso Program Given Data Parameters

Abstract

Compressed sensing (CS) is a paradigm in which a structured high-dimensional signal may be recovered from random, under-determined, and corrupted linear measurements. Lasso programs are effective for solving CS problems due to their proven ability to leverage underlying signal structure. Three popular Lasso programs are equivalent in a sense and sometimes used interchangeably. Tuned by a governing parameter, each admit an optimal parameter choice. For sparse or low-rank signal structures, this choice yields minimax order-optimal error. While CS is well-studied, existing theory for Lasso programs typically concerns this optimally tuned setting. However, the optimal parameter value for a Lasso program depends on properties of the data, and is typically unknown in practical settings. Performance in empirical problems thus hinges on a program’s parameter sensitivity: it is desirable that small variation about the optimal parameter choice begets small variation about the optimal risk. We examine the risk for these three programs and demonstrate that their parameter sensitivity can differ for the same data. We prove a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">gauge-constrained</i> Lasso program admits asymptotic cusp-like behaviour of its risk in the limiting low-noise regime. We prove that a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">residual-constrained</i> Lasso program has asymptotically suboptimal risk for very sparse vectors. These results contrast observations about an <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">unconstrained</i> Lasso program, which is relatively less sensitive to its parameter choice. We support the asymptotic theory with numerical simulations, demonstrating that parameter sensitivity of Lasso programs is readily observed for even modest dimensional parameters. Importantly, these simulations demonstrate regimes in which a Lasso program exhibits sensitivity to its parameter choice, though the other two do not. We hope this work aids practitioners in selecting a Lasso program for their problem.