&lt;inline-formula&gt; &lt;tex-math notation="LaTeX"&gt;$\ell _{p}$ &lt;/tex-math&gt;&lt;/inline-formula&gt;-Regularized Least Squares &lt;inline-formula&gt; &lt;tex-math notation="LaTeX"&gt;$(0&amp;lt;p&amp;lt;1)$ &lt;/tex-math&gt;&lt;/inline-formula&gt; and Critical Path

Masahiro Yukawa,Shun-Ichi Amari

doi:10.1109/tit.2015.2501362

Abstract

This paper elucidates the underlying structures of ℓ p -regularized least squares problems in the nonconvex case of 0 <; p <; 1. The difference between two formulations is highlighted (which does not occur in the convex case of p = 1): 1) an ℓ p -constrained optimization (P c p ) and 2) an ℓ p -penalized (unconstrained) optimization (L λ p ). It is shown that the solution path of (L λ p ) is discontinuous and also a part of the solution path of (P c p ). As an alternative to the solution path, a critical path is considered, which is a maximal continuous curve consisting of critical points. Critical paths are piecewise smooth, as can be seen from the viewpoint of the variational method, and generally contain non-optimal points, such as saddle points and local maxima as well as global/local minima. Our study reveals multiplicity (non-monotonicity) in the correspondence between the regularization parameters of (P c p ) and (L λ p ). Two particular paths of critical points connecting the origin and an ordinary least squares (OLS) solution are studied further. One is a main path starting at an OLS solution, and the other is a greedy path starting at the origin. Part of the greedy path can be constructed with a generalized Minkowskian gradient. This paper of greedy path leads to a nontrivial close-link between the optimization problem of ℓ p -regularized least squares and the greedy method of orthogonal matching pursuit.

Full Text