quasi-Newton Methods For Unconstrained Optimization Research Articles

Memoryless Quasi-Newton Methods for Unconstrained Optimization

Memoryless Quasi-Newton Methods for Unconstrained Optimization

An Overview of Some Practical Quasi-Newton Methods for Unconstrained Optimization

Quasi-Newton methods are among the most practical and efficient iterative methods for solving unconstrained minimization problems. In this paper we give an overview of some of these methods with focus primarily on the Hessian approximation updates and modifications aimed at improving their performance.

Numerical Experience with Multiple Update Quasi-Newton Methods for Unconstrained Optimization

The authors have derived what they termed quasi-Newton multi step methods in [2]. These methods have demonstrated substantial numerical improvements over the standard single step Secant-based BFGS. Such methods use a variant of the Secant equation that the updated Hessian (or its inverse) satisfies at each iteration. In this paper, new methods will be explored for which the updated Hessians satisfy multiple relations of the Secant-type. A rational model is employed in developing the new methods. The model hosts a free parameter which is exploited in enforcing symmetry on the updated Hessian approximation matrix thus obtained. The numerical performance of such techniques is then investigated and compared to other methods. Our results are encouraging and the improvements incurred supercede those obtained from other existing methods at minimal extra storage and computational overhead.

Global Convergence of the Broyden's Class of Quasi-Newton Methods with Nonmonotone Linesearch

In this paper, the Broyden class of quasi-Newton methods for unconstrained optimization is inves- tigated. Non-monotone linesearch procedure is introduced, which is combined with the Broyden's class. Under the convexity assumption on objective function, the global convergence of the Broyden's class is proved.

Implicit updates in multistep quasi-Newton methods

We consider multistep quasi-Newton methods for unconstrained optimization. These methods were introduced by Ford and Moghrabi [1,2], who showed how interpolating curves could be used to derive a generalization of the secant equation (the relation normally employed in the construction of quasi-Newton methods). One of the most successful of these multistep methods makes use of the current approximation to the Hessian to determine the parametrization of the interpolating curve in the variable-space and, hence, the generalized updating formula. In this paper, we investigate the use of implicit updates to the approximate Hessian, in an attempt to determine a better parametrization of the interpolation (while avoiding the computational burden of actually carrying out the update) and, thus, improve the numerical performance of such algorithms.

A nonlinear model for function-value multistep methods

We develop a framework employing scaling functions for the construction of multistep quasi-Newton methods for unconstrained optimization. These methods utilize values of the objective function. They are constructed via interpolants of the m+1 most recent iterates/gradient evaluations, and possess a free parameter which introduces an additional degree of flexibility. This permits the interpolating functions to assimilate information, in the form of function-values, which is readily available at each iteration. Motivated by previous experience [1] with the use of function-values in multistep methods, we investigate the incorporation of this information in the construction of the Hessian approximation at each iteration, in an attempt to accelerate convergence. We concentrate on a specific example from the general family of methods, corresponding to a particular choice of the scaling function, and from it derive three new algorithms. The relative numerical performance of these methods is assessed, and the most successful of them is then compared with the standard BFGS method and with an earlier algorithm utilizing function-values, also developed by the authors [1].

Reduced-Hessian Quasi-Newton Methods for Unconstrained Optimization

Quasi-Newton methods are reliable and efficient on a wide range of problems, but they can require many iterations if the problem is ill-conditioned or if a poor initial estimate of the Hessian is used. In this paper, we discuss methods designed to be more efficient in these situations. All the methods to be considered exploit the fact that quasi-Newton methods accumulate approximate second-derivative information in a sequence of expanding subspaces. Associated with each of these subspaces is a certain reduced approximate Hessian that provides a complete but compact representation of the second derivative information approximated up to that point. Algorithms that compute an explicit reduced-Hessian approximation have two important advantages over conventional quasi-Newton methods. First, the amount of computation for each iteration is significantly less during the early stages. This advantage is increased by forcing the iterates to linger on a manifold whose dimension can be significantly smaller than the subspace in which curvature has been accumulated. Second, approximate curvature along directions that lie off the manifold can be reinitialized as the iterations proceed, thereby reducing the influence of a poor initial estimate of the Hessian. These advantages are illustrated by extensive numerical results from problems in the CUTE test set. Our experiments provide strong evidence that reduced-Hessian quasi-Newton methods are more efficient and robust than conventional BFGS methods and some recently proposed extensions.

Metric-Based Parameterizations for Multi-Step Unconstrained Optimization

We consider multi-step quasi-Newton methods for unconstrained optimization. These methods were introduced by Ford and Moghrabi (Appl. Math., vol. 50, pp. 305–323, 1994s Optimization Methods and Software, vol. 2, pp. 357–370, 1993), who showed how interpolating curves could be used to derive a generalization of the Secant Equation (the relation normally employed in the construction of quasi-Newton methods). One of the most successful of these multi-step methods makes use of the current approximation to the Hessian to determine the parameterization of the interpolating curve in the variable-space and, hence, the generalized updating formula. In this paper, we investigate new parameterization techniques to the approximate Hessian, in an attempt to determine a better Hessian approximation at each iteration and, thus, improve the numerical performance of such algorithms.

Numerical Experience with a Class of Self-Scaling Quasi-Newton Algorithms

Self-scaling quasi-Newton methods for unconstrained optimization depend upon updating the Hessian approximation by a formula which depends on two parameters (say, τ and θ) such that τ = 1, θ = 0, and θ = 1 yield the unscaled Broyden family, the BFGS update, and the DFP update, respectively. In previous work, conditions were obtained on these parameters that imply global and superlinear convergence for self-scaling methods on convex objective functions. This paper discusses the practical performance of several new algorithms designed to satisfy these conditions.

Global and Superlinear Convergence of a Restricted Class of Self-Scaling Methods with Inexact Line Searches, for Convex Functions

This paper studies the convergence properties of algorithms belonging to the class of self-scaling (SS) quasi-Newton methods for unconstrained optimization. This class depends on two parameters, say t_k and τ_k, for which the choice τ_k=1 gives the Broyden family of unscaled methods, where t_k=1 corresponds to the well known DFP method. We propose simple conditions on these parameters that give rise to global convergence with inexact line searches, for convex objective functions. The q-superlinear convergence is achieved if further restrictions on the scaling parameter are introduced. These convergence results are an extension of the known results for the unscaled methods. Because the scaling parameter is heavily restricted, we consider a subclass of SS methods which satisfies the required conditions. Although convergence for the unscaled methods with t_k ≥ 1 is still an open question, we show that the global and superlinear convergence for SS methods is possible and present, in particular, a new SS-DFP method.

Alternating multi-step quasi-Newton methods for unconstrained optimization

We consider multi-step quasi-Newton methods for unconstrained optimization. These methods were introduced by the authors (Ford and Moghrabi [5, 6, 8]), who showed how an interpolating curve in the variable-space could be used to derive an appropriate generalization of the Secant Equation normally employed in the construction of quasi-Newton methods. One of the most successful of these multi-step methods employs the current approximation to the Hessian to determine the parametrization of the interpolating curve and, hence, the derivatives which are required in the generalized updating formula. However, certain approximations were found to be necessary in the process, in order to reduce the level of computation required (which must be repeated at each iteration) to acceptable levels. In this paper, we show how a variant of this algorithm, which avoids the need for such approximations, may be obtained. This is accomplished by alternating, on successive iterations, a single-step and a two-step method. The results of a series of experiments, which show that the new algorithm exhibits a clear improvement in numerical performance, are reported.

An efficient class of switching type algorithms in the Broyden family

In this paper, we extend the switching BFGS/DFP algorithm of Fletcher and the switching BFGS/SR1 algorithm of Al-Baali to a class of switching type algorithms proposed within the Broyden family of quasi-Newton methods for unconstrained optimization. We propose some members of this class, which switch among the BFGS, the SRI and other desirable methods from the preconvex class. The switching technique is made in a certain sense upon the basis of estimating the size of the eigenvalues of the Hessian approximation. The results on a set of standard test problems show that several switching methods improve over the BFGS method and work almost similarly to certain idealized methods.

Variational quasi-Newton methods for unconstrained optimization

In this paper, we propose new members of the Broyden family of quasi-Newton methods. We develop, on the basis of well-known least-change results for the BFGS and DFP updates, a measure for the Broyden family which seeks to take into account the change in both the Hessian approximation and its inverse. The proposal is then to choose the formula which gives the least value of this measure in terms of the two parameters available, and hence to produce an update which is optimal in the sense of the given measure. Several approaches to the problem of minimizing the measure are considered, from which new updates are obtained. In particular, one approach yields a new variational result for the Davidon optimally conditioned method and another yields a reasonable modification to this method. The paper is also concerned with the possibility of estimating, in a certain sense, the size of the eigenvalues of the Hessian approximation on the basis of two available scalars. This allows one to derive further modifications to the above-mentioned methods. Comparisons with the BFGS and Davidson methods are made on a set of standard test problems that show promising results for certain new methods.

Parallel quasi-Newton methods for unconstrained optimization

We discuss methods for solving the unconstrained optimization problem on parallel computers, when the number of variables is sufficiently small that quasi-Newton methods can be used. We concentrate mainly, but not exclusively, on problems where function evaluation is expensive. First we discuss ways to parallelize both the function evaluation costs and the linear algebra calculations in the standard sequential secant method, the BFGS method. Then we discuss new methods that are appropriate when there are enough processors to evaluate the function, gradient, and part but not all of the Hessian at each iteration. We develop new algorithms that utilize this information and analyze their convergence properties. We present computational experiments showing that they are superior to parallelization either the BFGS methods or Newton's method under our assumptions on the number of processors and cost of function evaluation. Finally we discuss ways to effectively utilize the gradient values at unsuccessful trial points that are available in our parallel methods and also in some sequential software packages.

Quasi-Newton Algorithms with Updates from the Preconvex Part of Broyden's Family

Most popular quasi-Newton methods for unconstrained optimization are from the one-parameter Broyden's family, especially from a subset of the family: the so-called convex class, i.e. members with the parameter 0≤φ≤1. In particular, the BFGS method corresponding to φ=0 is generally accepted as the best available. In this paper, we investigate the updates in Broyden's family that use negative parameter φ. We show that, for convex objective functions, these methods have the same global convergence property as BFGS and may also have superlinear convergence rate if φ≤0 is suitably chosen at each iteration. We present numerical evidence exhibiting that methods with φ≤ 0 can be more efficient than BFGS in terms of numbers of function evaluations and iterations. In particular, we compare two algorithms using φ≤0 with BFGS and show the relative merit of these algorithms.

Quasi-Newton Methods for Unconstrained Optimization

Quasi-Newton Methods for Unconstrained Optimization