Least-Squares Estimation, Linear Programming, and Momentum: A Geometric Parametrization of Local Minima

Abstract

In contrast to estimation by ordinary least squares, estimation by total least squares has much less favourable properties as far as existence and uniqueness of local minima is concerned. Indeed, as elementary examples show, and contrary to intuition gleaned from the Gauss-Markov theorem for ordinary least squares, for certain data sets this problem can have nonisolated local minima and local maxima. Using Morse theory and the Lie theory of coadjoint orbits, we show that, despite this apparent degeneracy, the distribution of critical points of least-squares problems is remarkably well behaved. For arbitrary data, the least-squares function is perfect in the sense of the Morse-Bott theory. In particular, the set of local minima always forms a connected manifold while there exists a unique minimum value.