Standard BFGS Method Research Articles

Global Convergence of a Modified Two-Parameter Scaled BFGS Method with Yuan-Wei-Lu Line Search for Unconstrained Optimization

The BFGS method is one of the most efficient quasi-Newton methods for solving small- and medium-size unconstrained optimization problems. For the sake of exploring its more interesting properties, a modified two-parameter scaled BFGS method is stated in this paper. The intention of the modified scaled BFGS method is to improve the eigenvalues structure of the BFGS update. In this method, the first two terms and the last term of the standard BFGS update formula are scaled with two different positive parameters, and the new value of yk is given. Meanwhile, Yuan-Wei-Lu line search is also proposed. Under the mentioned line search, the modified two-parameter scaled BFGS method is globally convergent for nonconvex functions. The extensive numerical experiments show that this form of the scaled BFGS method outperforms the standard BFGS method or some similar scaled methods.

A bilevel learning approach for optimal observation placement in variational data assimilation

In this paper we propose a bilevel optimization approach for the placement of space and time observations in variational data assimilation problems. Within the framework of supervised learning, we consider a bilevel problem where the lower-level task is the variational reconstruction of the initial condition of a semilinear system, and the upper-level problem solves the optimal placement with help of a sparsity inducing function. Due to the pointwise nature of the observations, an optimality system with regular Borel measures is obtained as necessary optimality condition for the lower-level problem. The latter is then considered as constraint for the upper-level instance, yielding an optimization problem constrained by a multi-state system with measures. We demonstrate existence of Lagrange multipliers and derive a necessary optimality system characterizing the optimal solutions of the bilevel problem. The numerical solution is carried out also on two levels. The lower-level problem is solved using a standard BFGS method, while the upper-level one is solved by means of a projected BFGS algorithm, based on the estimation of -active sets. Finally some numerical experiments are presented to illustrate the main features of our approach.

The global convergence of a modified BFGS method for nonconvex functions

The standard BFGS method plays an important role among the quasi-Newton algorithms for constrained/un-constrained optimization problems. However, Dai (2003) constructed a counterexample to demonstrate that this method may fail for non-convex functions with inexact Wolfe line searches, and Mascarenhas (2004) showed the nonconvergence of this method and other methods in the Broyden family, even with exact line search techniques. These works motivate us to try to find another way to obtain another quasi-Newton method with better convergence based on the standard BFGS formula. In this paper, four approaches are used in the designed algorithm: (1) a modified weak Wolfe–Powell line search technique is introduced; (2) if one defined condition is satisfied, then the search direction and its associated step length are accepted, and the next point is designed; (3) otherwise, a parabolic will be presented, and it is regarded as the projection surface to avoid using the failed direction, and the next point x k + 1 is generated by a projection technique; and (4) to easily obtain the global convergence of the proposed algorithm, the projection point is not used in the current BFGS update but rather for all the next iterations. The new algorithm possesses global convergence for general functions under the inexact modified weak Wolfe–Powell line search technique, and it is shown that other methods in the Broyden class also have this property. Numerical results are reported to show the performance of the given algorithm and other similar methods.

A new regularized limited memory BFGS-type method based on modified secant conditions for unconstrained optimization problems

In the framework of large-scale optimization problems, the standard BFGS method is not affordable due to memory constraints. The so-called limited-memory BFGS (L-BFGS) method is an adaption of the BFGS method for large-scale settings. However, the standard BFGS method and therefore the standard L-BFGS method only use the gradient information of the objective function and neglect function values. In this paper, we propose a new regularized L-BFGS method for solving large scale unconstrained optimization problems in which more available information from the function and gradient values are employed to approximate the curvature of the objective function. The proposed method utilizes a class of modified quasi-Newton equations in order to achieve higher order accuracy in approximating the second order curvature of the objective function. Under some standard assumptions, we provide the global convergence property of the new method. In order to provide an efficient method for finding global minima of a continuously differentiable function, a hybrid algorithm that combines a genetic algorithm (GA) with the new proposed regularized L-BFGS method has been proposed. This combination leads the iterates to a stationary point of the objective function with higher chance of being global minima. Numerical results show the efficiency and robustness of the new proposed regularized L-BFGS and its hybridized version with GA in practice.

A combined class of self-scaling and modified quasi-Newton methods

Techniques for obtaining safely positive definite Hessian approximations with self-scaling and modified quasi-Newton updates are combined to obtain `better' curvature approximations in line search methods for unconstrained optimization. It is shown that this class of methods, like the BFGS method, has the global and superlinear convergence for convex functions. Numerical experiments with this class, using the well-known quasi-Newton BFGS, DFP and a modified SR1 updates, are presented to illustrate some advantages of the new techniques. These experiments show that the performance of several combined methods are substantially better than that of the standard BFGS method. Similar improvements are also obtained if the simple sufficient function reduction condition on the steplength is used instead of the strong Wolfe conditions.

Spectral Scaling BFGS Method

In this paper, we scale the quasiNewton equation and propose a spectral scaling BFGS method. The method has a good selfcorrecting property and can improve the behavior of the BFGS method. Compared with the standard BFGS method, the single-step convergence rate of the spectral scaling BFGS method will not be inferior to that of the steepest descent method when minimizing an n-dimensional quadratic function. In addition, when the method with exact line search is applied to minimize an n-dimensional strictly convex function, it terminates within n steps. Under appropriate conditions, we show that the spectral scaling BFGS method with Wolfe line search is globally and R-linear convergent for uniformly convex optimization problems. The reported numerical results show that the spectral scaling BFGS method outperforms the standard BFGS method.

Sufficient descent directions in unconstrained optimization

Descent property is very important for an iterative method to be globally convergent. In this paper, we propose a way to construct sufficient descent directions for unconstrained optimization. We then apply the technique to derive a PSB (Powell-Symmetric-Broyden) based method. The PSB based method locally reduces to the standard PSB method with unit steplength. Under appropriate conditions, we show that the PSB based method with Armijo line search or Wolfe line search is globally and superlinearly convergent for uniformly convex problems. We also do some numerical experiments. The results show that the PSB based method is competitive with the standard BFGS method.

Enriched Algorithms for Large Scale Unconstrained Optimization

A new method for solving Large-Scale problems in the unconstrained optimization has been proposed in this research depending on the BFGS method. The limited memory is used in the BFGS method by multiplying the BFGS matrix by a vector to obtain vectors instead of matrices and only two vectors can be stored, by modifying the algorithm given by Nocedal J (1999). The purpose of this algorithm is to enable us to solving the Large-Scale Problems, as it is obvious to everyone that the computer can store millions of vectors, whereas its ability in storing matrices is limited. The present method in this research is applied on seven nonlinear functions in order to evaluate the method efficiency in the numbers of iterations (NOI), number of functions (NOF) and function value and comparing it with the standard BFGS method after updating. This method has been applied on functions with variables till 1000000 and more than that. From comparing the results, we fined that this algorithm was the best.

Three-step fixed-point quasi-Newtonmethods for unconstrained optimisation

Multistep quasi-Newton methods were introduced by Ford and Moghrabi [1]. They address the problem of the unconstrained minimisation of a function f : ℝ n → ℝ whose gradient and Hessian are denoted by g and G, respectively. These methods generalised the standard construction of quasi-Newton methods and were based on employing interpolatory polynomials to utilise information from more than one previous step. In a series of papers, Ford and Moghrabi [2–5] have developed various techniques for determining the parametrisation of the interpolating curves. In [2], they introduced two-step metric-based methods which determine the set of parameter values required through measuring distances between various pairs of the iterates employed in the current interpolation. One of the most successful methods in [2] was found to be in the “fixed-point” class, in which the parametrisation of the interpolating curve is determined, at each iteration, by reference to distances measured from a fixed iterate. As suggested in [1], multistep quasi-Newton methods can be constructed for any number of steps.In this paper, we therefore extend the previous work by describing the development of some three-step methods which use the “fixed-point” approach and data derived from the latest four iterates. The experimental results provide evidence that the new methods offer a significant improvement in performance when compared with the standard BFGS method and the unit-spaced three-step method, particularly as the dimension of the test problems grows.

A nonlinear model for function-value multistep methods

We develop a framework employing scaling functions for the construction of multistep quasi-Newton methods for unconstrained optimization. These methods utilize values of the objective function. They are constructed via interpolants of the m+1 most recent iterates/gradient evaluations, and possess a free parameter which introduces an additional degree of flexibility. This permits the interpolating functions to assimilate information, in the form of function-values, which is readily available at each iteration. Motivated by previous experience [1] with the use of function-values in multistep methods, we investigate the incorporation of this information in the construction of the Hessian approximation at each iteration, in an attempt to accelerate convergence. We concentrate on a specific example from the general family of methods, corresponding to a particular choice of the scaling function, and from it derive three new algorithms. The relative numerical performance of these methods is assessed, and the most successful of them is then compared with the standard BFGS method and with an earlier algorithm utilizing function-values, also developed by the authors [1].

Extra updates for the bfgs method*

This paper considers employing extra updates for the BFGS method for unconstrained optimization. The usual BFGS Hessian is updated a number of times, depending on the information of the first order derivatives, to obtain a new Hessian approximation at each iteration. Two approaches are proposed. One of them has the same properties of global and superlinear convergence on convex functions the BFGS method has, and another has the same property of quadratic termination without exact line searches that the symmetric rank-one method has. The new algorithms attempt to combine the best features of certain methods which are intended for either parallel computation or large scale optimization. It is concluded that some new algorithms are competitive with the standard BFGS method

Modifying the BFGS method

We propose a modified BFGS method and study the global and superlinear convergence properties of this method. We show that under certain circumstances this modified BFGS method has a better mechanism of correcting the eigenvalues. This modified method can also be regarded as an implementation technique of the standard BFGS method.

Minimum curvature multistep quasi-Newton methods

We consider multistep quasi-Newton methods for unconstrained optimization problems. Such methods were introduced by the authors [1–3], and employ interpolating polynomials to utilize data from the m most recent iterations. For the case m = 2, we observe that there is a free parameter (used in constructing the interpolation) which is essentially at our disposal. We propose (as a criterion for determining this parameter) the minimization of a measure of the curvature of the interpolating polynomial, thus producing a “smooth” interpolant. We show how this “minimum curvature” problem may be solved cheaply and effectively at each iteration. The performance of an algorithm which employs this technique is compared (numerically) with that of the standard BFGS method and of previously-introduced multistep methods [2,3].

A quasi-Newton method using a nonquadratic model

Most “quasi-Newton” methods in common use for function minimisation use a quadratic model to approximate the underlying objective function at the current estimate of the minimum. In this paper we describe new minimisation algorithms derived by replacing the quadratic model with a polynomial type model involving a free parameter which is determined implicitly by means of information about the objective function. We show how this information may be efficiently utilised in an optimisation method of quasi-Newton type. Numerical tests were carried out to demonstrate the performance of the new algorithms in comparison to the standard BFGS method.