Nonmonotone Line Search Research Articles

A new filter QP-free method for the nonlinear inequality constrained optimization problem

In this paper, a filter QP-free infeasible method with nonmonotone line search is proposed for minimizing a smooth optimization problem with smooth inequality constraints. This proposed method is based on the solution of nonsmooth equations, which are obtained by the Lagrangian multiplier method and the function of the nonlinear complementarity problem for the Karush–Kuhn–Tucker optimality conditions. Especially, each iteration of this method can be viewed as a perturbation of a Newton or quasi-Newton iteration on both the primal and dual variables for the solution of the Karush–Kuhn–Tucker optimality conditions. What is more, it is considered to use the function of the nonlinear complementarity problem in the filter, which makes the proposed algorithm avoid the incompatibility. Then the global convergence of the proposed method is given. And under some mild conditions, the superlinear convergence rate can be obtained. Finally, some preliminary numerical results are shown to illustrate that the proposed filter QP-free infeasible method is quite promising.

A structured diagonal Hessian approximation method with evaluation complexity analysis for nonlinear least squares

This work proposes a Jacobian-free strategy for addressing large-scale nonlinear least-squares problems, in which structured secant conditions are used to define a diagonal approximation for the Hessian matrix. Proper safeguards are devised to ensure descent directions along the generated sequence. Worst-case evaluation analysis is provided within the framework of a non-monotone line search. Numerical experiments contextualize the proposed strategy, by addressing structured problems from the literature, also solved by related and recently presented conjugate gradient and multivariate spectral gradient strategies, as well as the classic Fletcher–Reeves conjugate gradient, and the Raydan–Barzilai–Borwein methods. The comparative computational results show a favorable performance of the proposed approach, mainly as far as robustness is concerned.

Multipoint secant and interpolation methods with nonmonotone line search for solving systems of nonlinear equations

Multipoint secant and interpolation methods are effective tools for solving systems of nonlinear equations. They use quasi-Newton updates for approximating the Jacobian matrix. Owing to their ability to more completely utilize the information about the Jacobian matrix gathered at the previous iterations, these methods are especially efficient in the case of expensive functions. They are known to be local and superlinearly convergent. We combine these methods with the nonmonotone line search proposed by Li and Fukushima (2000), and study global and superlinear convergence of this combination. Results of numerical experiments are presented. They indicate that the multipoint secant and interpolation methods tend to be more robust and efficient than Broyden’s method globalized in the same way.

Spectral projected gradient method for stochastic optimization

We consider the Spectral Projected Gradient method for solving constrained optimization problems with the objective function in the form of mathematical expectation. It is assumed that the feasible set is convex, closed and easy to project on. The objective function is approximated by a sequence of different Sample Average Approximation functions with different sample sizes. The sample size update is based on two error estimates—SAA error and approximate solution error. The Spectral Projected Gradient method combined with a nonmonotone line search is used. The almost sure convergence results are achieved without imposing explicit sample growth condition. Preliminary numerical results show the efficiency of the proposed method.

A Levenberg–Marquardt Method for Nonlinear Complementarity Problems Based on Nonmonotone Trust Region and Line Search Techniques

Using the FB function, we propose a new Levenberg–Marquardt algorithm for nonlinear complementarity problem. To obtain the global convergence, the algorithm uses the nonmonotone trust region and line search techniques under a convenient boundedness assumption. Furthermore, we get local superlinear/quadratic convergence of the algorithm under a nonsingularity condition. Some numerical examples are given to illustrate the performance and efficiency of the presented algorithm.

A nonmonotone smoothing Newton algorithm for solving general box constrained variational inequalities

In this paper, based on a new smoothing function, the general box constrained variational inequalities are solved by a smoothing Newton algorithm with a nonmonotone line search. The proposed algorithm is proved to be globally and locally superlinearly convergent under suitable assumptions. The preliminary numerical results are reported.

On the convergence rate of scaled gradient projection method

The scaled gradient projection (SGP) method, which can be viewed as a promising improvement of the classical gradient projection method, is a quite efficient solver for real-world problems arising in image science and machine learning. Most recently, Bonettini and Prato [Inverse Probl. 2015;31:095008. 20 p] proved that the SGP method with the monotone Armijo line search technique has the convergence rate, where k counts the iteration. In this paper, we first show that the SGP method could be equipped with the nonmonotone line search procedure proposed by Zhang and Hager [SIAM J Optim. 2004;14:1043–1056]. To some extent, such a nonmonotone technique might improve the performance of SGP method, because its effectiveness has been verified for unconstrained optimization by comparing with the traditional monotone and nonmonotone strategies. Then, we prove that the new SGP method also has the convergence rate under the condition that the objective function is convex. Furthermore, we derive the linear convergence of the SGP algorithm under the strongly convexity assumption of the involved objective function.

Gradient method with multiple damping for large-scale unconstrained optimization

Gradient methods are popular due to the fact that only gradient of the objective function is required. On the other hand, the methods can be very slow if the objective function is very ill-conditioned. One possible reason for the inefficiency of the gradient methods is that a constant criterion, which aims only at reducing the function value, has been used in choosing the steplength, and this leads to a stable dynamic system giving slow convergence. To overcome this, we propose a new gradient method with multiple damping, which works on the objective function and the norm of the gradient vector simultaneously. That is, the proposed method is constructed by combining damping with line search strategies, in which an individual adaptive parameter is proposed to damp the gradient vector while line searches are used to reduce the function value. Global convergence of the proposed method is established under both backtracking and nonmonotone line search. Finally, numerical results show that the proposed algorithm performs better than some well-known CG-based methods.

A new regularized quasi-Newton method for unconstrained optimization

In this paper, we propose a new regularized quasi-Newton method for unconstrained optimization. At each iteration, a regularized quasi-Newton equation is solved to obtain the search direction. The step size is determined by a non-monotone Armijo backtracking line search. An adaptive regularized parameter, which is updated according to the step size of the line search, is employed to compute the next search direction. The presented method is proved to be globally convergent. Numerical experiments show that the proposed method is effective for unconstrained optimizations and outperforms the existing regularized Newton method.

Quasi-Newton methods for constrained nonlinear systems: complexity analysis and applications

We address the solution of constrained nonlinear systems by new linesearch quasi-Newton methods. These methods are based on a proper use of the projection map onto the convex constraint set and on a derivative-free and nonmonotone linesearch strategy. The convergence properties of the proposed methods are presented along with a worst-case iteration complexity bound. Several implementations of the proposed scheme are discussed and validated on bound-constrained problems including gas distribution network models. The results reported show that the new methods are very efficient and competitive with an existing affine-scaling procedure.

A non-monotone linear search algorithm with mixed direction on Stiefel manifold

In this paper, we propose a non-monotone line search method for solving optimization problems on Stiefel manifold. The main novelty of our approach is that our method uses a search direction based on a linear combination of descent directions and a Barzilai–Borwein line search. The feasibility is guaranteed by projecting each iterate on the Stiefel manifold through SVD (singular value decomposition) factorizations. Some theoretical results for analysing the algorithm are presented. Finally, we provide numerical experiments for comparing our algorithm with other state-of-the-art procedures. The code is available online. The experimental results show that the proposed algorithm is competitive with other approaches and for particular problems, the computational performance is better than the state-of-the-art algorithms.

A nonmonotone line search method for stochastic optimization problems

A nonmonotone line search method for solving unconstrained optimization problems with the objective function in the form of mathematical expectation is proposed and analyzed. The method works with approximate values of the objective function obtained with increasing sample sizes and improves accuracy gradually. Nonmonotone rule significantly enlarges the set of admissible search directions and prevents unnecessarily small steps at the beginning of the iterative procedure. The convergence is shown for any search direction that approaches the negative gradient in the limit. The convergence results are obtained in the sense of zero upper density. Initial numerical results confirm theoretical results and show efficiency of the proposed approach.

Forward-Backward Envelope for the Sum of Two Nonconvex Functions: Further Properties and Nonmonotone Linesearch Algorithms

We propose ZeroFPR, a nonmonotone linesearch algorithm for minimizing the sum of two nonconvex functions, one of which is smooth and the other possibly nonsmooth. ZeroFPR is the first algorithm that, despite being fit for fully nonconvex problems and requiring only the black-box oracle of forward-backward splitting (FBS) --- namely evaluations of the gradient of the smooth term and of the proximity operator of the nonsmooth one --- achieves superlinear convergence rates under mild assumptions at the limit point when the linesearch directions satisfy a Dennis-Mor\'e condition, and we show that this is the case for quasi-Newton directions. Our approach is based on the forward-backward envelope (FBE), an exact and strictly continuous penalty function for the original cost. Extending previous results we show that, despite being nonsmooth for fully nonconvex problems, the FBE still enjoys favorable first- and second-order properties which are key for the convergence results of ZeroFPR. Our theoretical results are backed up by promising numerical simulations. On large-scale problems, by computing linesearch directions using limited-memory quasi-Newton updates our algorithm greatly outperforms FBS and its accelerated variant (AFBS).

A variant nonmonotone smoothing algorithm with improved numerical results for large-scale LWCPs

This paper considers the linear weighted complementarity problem (denoted by LWCP) introduced by Potra (SIAM J Optim 22:1634–1654, 2012). Based on two weighted smoothing functions, we propose a new nonmonotone smoothing algorithm for solving the LWCP and establish its global and local quadratic convergence without the strict complementarity assumption. Compared to existing nonmonotone smoothing algorithms, the proposed algorithm solves the linear system only approximately which can save the computation work when one solves large-scale LWCPs. Moreover, the nonmonotone line search technique adopted in this paper includes the usual monotone line search and some existing nonmonotone line searches as special cases. Numerical results show that our algorithm is considerably efficient for solving large-scale LWCPs.

A new subspace minimization conjugate gradient method with nonmonotone line search for unconstrained optimization

A new subspace minimization conjugate gradient algorithm with a nonmonotone Wolfe line search is proposed and analyzed. In the scheme, we propose two choices of the search direction by minimizing a...

A QP-free algorithm without a penalty function or a filter for nonlinear general-constrained optimization

In this paper, we present a QP-free algorithm without a penalty function or a filter for nonlinear general-constrained optimization. At each iteration, three systems of linear equations with the same coefficient matrix are solved to yield search direction; the nonmonotone line search ensures that the objective function or constraint violation function is sufficiently reduced. There is no feasibility restoration phase in our algorithm, which is necessary for filter methods. The algorithm possesses global convergence as well as superlinear convergence under some mild conditions including a weaker assumption of positive definiteness. Finally, some preliminary numerical results are reported.

An extended nonmonotone line search technique for large-scale unconstrained optimization

In this paper, an extended nonmonotone line search is proposed to improve the efficiency of the existing line searches. This line search is first proved to be an extension of the classical line search rules. On the one hand, under mild assumptions, global convergence and R-linear convergence are established for the new line search rule. On the other hand, by numerical experiments, it is shown that the line search can integrate the advantages of the existing methods in searching for a suitable step-size. Combined with the spectral step-size, a class of spectral gradient algorithms are developed and employed to solve a large number of benchmark test problems from CUTEst. Numerical results show that the new line search is promising in solving large-scale optimization problems, and outperforms the other similar ones as it is combined with a spectral gradient method.

An adaptive nonmonotone trust region algorithm

Based on an eigenvalue analysis conducted on the scaled memoryless quasi-Newton updating formulas BFGS and DFP, an adaptive choice for the trust region radius is proposed. Then, using a trust region ratio obtained from a nonmonotone line search strategy, an adaptive nonmonotone trust region algorithm is developed. Under proper conditions, it is briefly shown that the proposed algorithm is globally and locally superlinearly convergent. Numerical experiments are done on a set of unconstrained optimization test problems of the CUTEr collection, using the Dolan–Moré performance profile. They show efficiency of the proposed algorithm.

Smoothing Newton algorithm for the circular cone programming with a nonmonotone line search

AbstractIn this paper, we present a nonmonotone smoothing Newton algorithm for solving the circular cone programming (CCP) problem in which a linear function is minimized or maximized over the intersection of an affine space with the circular cone. Based on the relationship between the circular cone and the second-order cone (SOC), we reformulate the CCP problem as the second-order cone problem (SOCP). By extending the nonmonotone line search for unconstrained optimization to the CCP, a nonmonotone smoothing Newton method is proposed for solving the CCP. Under suitable assumptions, the proposed algorithm is shown to be globally and locally quadratically convergent. Some preliminary numerical results indicate the effectiveness of the proposed algorithm for solving the CCP.

A modified nonmonotone BFGS algorithm for unconstrained optimization

In this paper, a modified BFGS algorithm is proposed for unconstrained optimization. The proposed algorithm has the following properties: (i) a nonmonotone line search technique is used to obtain the step size alpha_{k} to improve the effectiveness of the algorithm; (ii) the algorithm possesses not only global convergence but also superlinear convergence for generally convex functions; (iii) the algorithm produces better numerical results than those of the normal BFGS method.