Inexact Line Search Research Articles

An adaptive conjugate gradient learning algorithm for efficient training of neural networks

An adaptive conjugate gradient learning algorithm has been developed for training of multilayer feedforward neural networks. The problem of arbitrary trial-and-error selection of the learning and momentum ratios encountered in the momentum backpropagation algorithm is circumvented in the new adaptive algorithm. Instead of constant learning and momentum ratios, the step length in the inexact line search is adapted during the learning process through a mathematical approach. Thus, the new adaptive algorithm provides a more solid mathematical foundation for neural network learning. The algorithm has been implemented in C on a SUN-SPARCstation and applied to two different domains: engineering design and image recognition. It is shown that the adaptive neural networks algorithm has superior convergence property compared with the momentum backpropagation algorithm.

A globally convergent primal—dual interior point algorithm for convex programming

In this paper, we study the global convergence of a large class of primal--dual interior point algorithms for solving the linearly constrained convex programming problem. The algorithms in this class decrease the value of a primal--dual potential function and hence belong to the class of so-called potential reduction algorithms. An inexact line search based on Armijo stepsize rule is used to compute the stepsize. The directions used by the algorithms are the same as the ones used in primal--dual path following and potential reduction algorithms and a very mild condition on the choice of the "centering parameter" is assumed. The algorithms always keep primal and dual feasibility and, in contrast to the polynomial potential reduction algorithms, they do not need to drive the value of the potential function towards -- ź in order to converge. One of the techniques used in the convergence analysis of these algorithms has its root in nonlinear unconstrained optimization theory.

Analysis of a self-scaling quasi-Newton method

We study the self-scaling BFGS method of Oren and Luenberger (1974) for solving unconstrained optimization problems. For general convex functions, we prove that the method is globally convergent with inexact line searches. We also show that the directions generated by the self-scaling BFGS method approach Newton's direction asymptotically. This would ensure superlinear convergence if, in addition, the search directions were well-scaled, but we show that this is not always the case. We find that the method has a major drawback: to achieve superlinear convergence it may be necessary to evaluate the function twice per iteration, even very near the solution. An example is constructed to show that the step-sizes required to achieve a superlinear rate converge to 2 and 0.5 alternately.

A counterexample of the convergence of Rosen's algorithm with inexact line search

Du and Zhang (1989) recently proved that Rosen's gradient projection method converges when using either exact or Armijo-type inexact line searches. We show here that the method fails to converge when the inexact line search proposed in the original paper is used. The same quadratic programming counterexample can be used to show that Ritter's method with Rosen's inexact line search also fails to converge.

Complexity analysis of a linear complementarity algorithm based on a Lyapunov function

We consider a path following algorithm for solving linear complementarity problems with positive semi-definite matrices. This algorithm can start from any interior solution and attain a linear rate of convergence. Moreover, if the starting solution is appropriately chosen, this algorithm achieves a complexity of O( $$\sqrt m$$ L}) iterations, wherem is the number of variables andL is the size of the problem encoding in binary. We present a simple complexity analysis for this algorithm, which is based on a new Lyapunov function for measuring the nearness to optimality. This Lyapunov function has itself interesting properties that can be used in a line search to accelerate convergence. We also develop an inexact line search procedure in which the line search stepsize is obtainable in a closed form. Finally, we extended this algorithm to handle directly variables which are unconstrained in sign and whose corresponding matrix is positive definite. The rate of convergence of this extended algorithm is shown to be independent of the number of such variables.

A Hybrid Variable Metric Update for the Recursive Quadratic Programming Method

The Recursive Quadratic Programming (RQP) method has become known as one of the most effective and efficient algorithms for solving engineering optimization problems. The RQP method uses variable metric updates to build approximations of the Hessian of the Lagrangian. If the approximation of the Hessian of the Lagrangian converges to the true Hessian of the Lagrangian, then the RQP method converges quadratically. The choice of a variable metric update has a direct effect on the convergence of the Hessian approximation. Most of the research performed with the RQP method uses some modification of the Broyden-Fletcher-Shanno (BFS) variable metric update. This paper describes a hybrid variable metric update that yields good approximations to the Hessian of the Lagrangian. The hybrid update combines the best features of the Symmetric Rank One and BFS updates, but is less sensitive to inexact line searches than the BFS update, and is more stable than the SR1 update. Testing of the method shows that the efficiency of the RQP method is unaffected by the new update but more accurate Hessian approximations are produced. This should increase the accuracy of the solutions obtained with the RQP method, and more importantly, provide more reliable information for post optimality analyses, such as parameter sensitivity studies.

Relaxation Methods for Problems with Strictly Convex Costs and Linear Constraints

Consider the problem of minimizing a strictly convex (possibly nondifferentiable and nonseparable) cost subject to linear constraints. We propose a dual coordinate ascent method for this problem that uses inexact line search and either essentially cyclic or Gauss-Southwell order of coordinate relaxation. We show, under very weak conditions, that this method generates a sequence of primal vectors converging to the optimal primal solution. Under an additional regularity assumption, and assuming that the effective domain of the cost function is polyhedral, we show that a related sequence of dual vectors converges in cost to the optimal cost. If the constraint set has an interior point in the effective domain of the cost function, then this sequence of dual vectors is bounded and each of its limit point(s) is an optimal dual solution. When the cost function is strongly convex, we show that the order of coordinate relaxation can become progressively more chaotic. These results significantly improve upon those obtained previously.

Efficient generalized conjugate gradient algorithms, part 1: Theory

The effect of inexact line search on conjugacy is studied in unconstrained optimization. A generalized conjugate gradient method based on this effect is proposed and shown to have global convergence for a twice continuously differentiable function with a bounded level set.

Conjugate gradient methods using quasi-Newton updates with inexact line searches

Conjugate gradient methods are conjugate direction or gradient deflection methods which lie somewhere between the method of steepest descent and Newton's method. Their principal advantage is that they do not require the storage of any matrices as in Newton's method, or as in quasi-Newton methods, and they are designed to converge faster than the method of steepest descent. Unlike quasi-Newton or variable-metric methods, these are fixed-metric methods in which the search direction at each iteration is based on an approximation to the inverse Hessian constructed by updating a fixed, symmetric, positive definite matrix, typically the identity matrix. The resulting approximation is usually not symmetric, although some variants force symmetry and hence derive memoryless quasi-Newton methods. In this paper, we present a scaled modified version of the conjugate gradient method suggested by Perry, which employs the quasi-Newton condition rather than conjugacy under inexact line searches, in order to derive the search directions. The analysis is extended to the memoryless quasi-Newton modification of this method, as suggested by Shanno. Computational experience on standard test problems indicates that the proposed method, along with Beale and Powell's restarts, improves upon existing conjugate gradient strategies.

Curved search methods for unconstrained optimization

The paper introduces a family of algorithms for unconstrained minimization of an nvariable twice continuously differentiate function /. Unlike classical met hods, which improve a current solution by moving along a straight line, the new methods improve the solution by moving along a quadratic curve in R nThe specific curve is determined by minimizing an appropriate model of f The algorithms thus obtained (called Curved Searchalgorithms) all possess a global convergence property combined with a quadratic rate of convergence. They are all using the same information and employing the same computational effort as the Newton method, which is in fact a member of this class, versions of cured search methods with inexact line search of the Goldstein-type are studied as well, retaining the above desirable convergence properties. We also discuss a version, called β-method not requiring a line search altogether. Computational experience reported in the paper points out to the potential improvement, which may be gain...

A Note on Properties of Iterative Procedures of Asymptotic Inference

This note deals with the article ‘On iterative procedures of asymptotic inference’ by K.O. DZHAPARIDZE (1983), in which an informal discussion is given on performing an unconstrained maximization or solving non–linear equations of statistics by iterative methods with the quadratic termination property. It discusses the theorem that if a maximized function, e.g. the likelihood function, is asymptotically quadratic, then for asymptotically efficient inference finitely many iterations are needed.It is argued here that the theory still applies if certain well specified inexact (hence computationally cheaper) line searches are used in the optimization.

A stable, rapidly converging conjugate gradient method for energy minimization

AbstractWe apply Shanno's conjugate gradient algorithm to the problem of minimizing the potential energy function associated with molecular mechanical calculations. Shanno's algorithm is stable with respect to roundoff errors and inexact line searches and converges rapidly to a minimum. Equally important, this algorithm can improve the rate of convergence to a minimum by a factor of 5 relative to Fletcher‐Reeves or Polak‐Ribière minimizers when used within the molecular mechanics package AMBER. Comparable improvements are found for a limited number of simulations when the Polak‐Ribière direction vector is incorporated into the Shanno algorithm.

A direct method of linearization for continuous minimax problems

We consider the problem of minimizing a nondifferentiable function that is the pointwise maximum over a compact family of continuously differentiable functions. We suppose that a certain convex approximation to the objective function can be evaluated. An iterative method is given which uses as successive search directions approximate solutions of semi-infinite quadratic programming problems calculated via a new generalized proximity algorithm. Inexact line searches ensure global convergence of the method to stationary points.

A method for minimizing the sum of a convex function and a continuously differentiable function

This paper presents a method for minimizing the sum of a possibly nonsmooth convex function and a continuously differentiable function. As in the convex case developed by the author, the algorithm is a descent method which generates successive search directions by solving quadratic programming subproblems. An inexact line search ensures global convergence of the method to stationary points.

Analogues of Dixon’s and Powell’s Theorems for Unconstrained Minimization with Inexact Line Searches

Dixon’s theorem (Math. Programming, 2 (1972), PP. 383–387) states that all variable metric methods in the Broyden class develop identical iterates when line searches are exact. Powell’s theorem (Rep. TP 495, AERE, Harwell, England, 1972) is a variant on this, which states that under similar conditions, the Hessian approximation developed by a BFGS update at any step is independent of the updates used at earlier steps. By modifying the way in which search directions are defined, we show how to remove the restrictive assumption on line searches in these two theorems. We show also that the BFGS algorithm, modified in this way, is equivalent to the three-term-recurrence (TTR) method on quadratic functions. Algorithmic implications are discussed and the results of some numerical experimentation are reported.

Descent Property and Global Convergence of the Fletcher—Reeves Method with Inexact Line Search

If an inexact lilne search which satisfies certain standard conditions is used . then it is proved that the Fletcher-Reeves method had a descent property and is globally convergent in a certain sense.

Use of a nonquadratic model in a conjugate-gradient method of optimization with inexact line searches

An algorithm for unconstrained minimization is developed which uses a rational function model, rather than a quadratic, as a basis for conjugate directions. The algorithm is similar to one previously proposed by the authors, but inexact linear searches are investigated in the present paper.

A least-distance programming procedure for minimization problems under linear constraints

In this paper, an algorithm is developed for solving a nonlinear programming problem with linear contraints. The algorithm performs two major computations. First, the search vector is determined by projecting the negative gradient of the objective function on a polyhedral set defined in terms of the gradients of the equality constraints and the near binding inequality constraints. This least-distance program is solved by Lemke's complementary pivoting algorithm after eliminating the equality constraints using Cholesky's factorization. The second major calculation determines a stepsize by first computing an estimate based on quadratic approximation of the function and then finalizing the stepsize using Armijo's inexact line search.

Convergence Conditions for Ascent Methods

Convergence Conditions for Ascent Methods