Least Squares, Modeling, and Signal Processing

Abstract

The concept of least squares (LS) as applied to an inconsistent system of linear or nonlinear equations is a fundamental tool in numerical analysis. Furthermore, these techniques have been applied with much success in solving many of the more challenging problems found in signal processing. In the standard LS problem, one seeks a choice for the vector x governing the inconsistent system of linear equations A x ≍ y so that these equations are best approximated in the least-squares error sense. In this paper, the concepts underlying an LS solution approach are presented in a tutorial fashion and only a basic knowledge of Euclidean spaces R n and C n is presumed. Much of the analysis is related to a linear system of equations where use of the fundamental fact that all Hermitian matrices have a full set of pairwise orthonormal eigenvectors plays a central role. Once the basic LS solution characterization of a linear system of equations has been made, a statistical analysis of this solution is undertaken. Conditions under which the LS solution is particularly sensitive to additive noise are established. This sensitivity can be decreased by using the concept of reduced rank approximation where a trade-off between estimation bias and estimation variance is made. The notion of linear least-squares error is then generalized to consider the case whereby the inconsistent system of linear equations A( θ ) x = y has a system matrix A( θ ) that depends on a set of real parameters θ . It is now desired to select both x and θ so as to obtain a best approximate solution. This is shown to lead to a modified LS solution. An important extension of this problem is next made whereby the multiple inconsistent system of linear equations A ( θ ) x k = y k for 1 ≤ k ≤ N is best approximated in the least-squares error sense. These concepts are then further generalized to include the task of finding an LS solution to a system of nonlinear equations described by F ( x , θ ) = y .