The solution of nonlinear inverse problems and the Levenberg-Marquardt method

Abstract

Although the Levenberg-Marquardt damped least-squares method is an extremely powerful tool for the iterative solution of nonlinear problems, its theoretical basis has not been described adequately in the literature. This is unfortunate, because Levenberg and Marquardt approached the solution of nonlinear problems in different ways and presented results that go far beyond the simple equation that characterizes the method. The idea of damping the solution was introduced by Levenberg, who also showed that it is possible to do that while at the same time reducing the value of a function that must be minimized iteratively. This result is not obvious, although it is taken for granted. Moreover, Levenberg derived a solution more general than the one currently used. Marquardt started with the current equation and showed that it interpolates between the ordinary least-squares-method and the steepest-descent method. In this tutorial, the two papers are combined into a unified presentation, which will help the reader gain a better understanding of what happens when solving nonlinear problems. Because the damped least-squares and steepest-descent methods are intimately related, the latter is also discussed, in particular in its relation to the gradient. When the inversion parameters have the same dimensions (and units), the direction of steepest descent is equal to the direction of minus the gradient. In other cases, it is necessary to introduce a metric (i.e., a definition of distance) in the parameter space to establish a relation between the two directions. Although neither Levenberg nor Marquardt discussed these matters, their results imply the introduction of a metric. Some of the concepts presented here are illustrated with the inversion of synthetic gravity data corresponding to a buried sphere of unknown radius and depth. Finally, the work done by early researchers that rediscovered the damped least-squares method is put into a historical context.