Chapter 16 - Least-Squares Problems

Abstract

This chapter presents linear least-squares. The normal equations are motivated by a figure showing that b − Ax should be normal to Ax. It is then shown that if m ≥ n, there is a unique least-squares solution if and only if x satisfies the normal equations. Also, x is unique if and only if A has full rank. The pseudoinverse or the Moore-Penrose generalized inverse is presented, and the condition number of an m × n matrix m ≥ n is defined using the pseudoinverse. There are three basic techniques for solving the overdetermined least-squares problem, m ≥ n, solving the normal equations, using the reduced QR decomposition, and using the reduced SVD. The most commonly used is the QR decomposition. The solution using the normal equations depends on the square of the condition number and should only be used with a well-conditioned matrix. The SVD approach generally is slower than QR but still effective. Both the QR and SVD algorithms are stable. There is a brief discussion of least-squares conditioning. If the angle between b and Ax is small, the conditioning depends on the condition number of A. If the angle is not close to π2 but the condition number is large, then sensitivity depends on the square of the condition number. If the angle is close to π2, the solution is near zero, but the errors are very large. For m ≥ n, the rank-deficient least-squares problem is presented and solved using the SVD. Among infinitely many solutions, the algorithm finds the one with minimum norm. The chapter ends with a discussion of underdetermined least-squares problems, m < n. An algorithm is developed that uses the QR decomposition applied to AT.