Trust Region Method For Unconstrained Optimization Research Articles

Solving the Cubic Regularization Model by a Nested Restarting Lanczos Method

As a variant of the classical trust-region method for unconstrained optimization, the cubic regularization of the Newton method introduces a cubic regularization term in the surrogate objective to adaptively adjust the updating step and deals with cases with both indefinite and definite Hessians. It has been demonstrated that the cubic regularization of the Newton method enjoys a good global convergence and is an efficient solver for the unconstrained minimization. The main computational cost in each iteration is to solve a cubic regularization subproblem. The Newton iteration is a common and efficient method for this task, especially for small- to medium-size problems. For large size problems, a Lanczos type method was proposed in [C. Cartis, N. I. M. Gould, and P. L. Toint, Math. Program., 127 (2011), pp. 245--295]. This method relies on a Lanczos procedure to reduce the large-scale cubic regularization subproblem to a small one and solve it by the Newton iteration. For large and ill-conditioned problems, the Lanczos method still needs to produce a large dimensional subspace to achieve a relatively highly accurate approximation, which declines its performance overall. In this paper, we first show that the cubic regularization subproblem can be equivalently transformed into a quadratic eigenvalue problem, which provides an eigensolver alternative to the Newton iteration. We then establish the convergence of the Lanczos method and also propose a nested restarting version for the large scale and ill-conditioned case. By integrating the nested restarting Lanczos iteration into the cubic regularization of the Newton method, we verify its efficiency for solving large scale minimization problems in CUTEst collection.

A non-monotone pattern search approach for systems of nonlinear equations

ABSTRACTIn this paper, a new pattern search is proposed to solve the systems of nonlinear equations. We introduce a new non-monotone strategy which includes a convex combination of the maximum function of some preceding successful iterates and the current function. First, we produce a stronger non-monotone strategy in relation to the generated strategy by Gasparo et al. [Nonmonotone algorithms for pattern search methods, Numer. Algorithms 28 (2001), pp. 171–186] whenever iterates are far away from the optimizer. Second, when iterates are near the optimizer, we produce a weaker non-monotone strategy with respect to the generated strategy by Ahookhosh and Amini [An efficient nonmonotone trust-region method for unconstrained optimization, Numer. Algorithms 59 (2012), pp. 523–540]. Third, whenever iterates are neither near the optimizer nor far away from it, we produce a medium non-monotone strategy which will be laid between the generated strategy by Gasparo et al. [Nonmonotone algorithms for pattern search methods, Numer. Algorithms 28 (2001), pp. 171–186] and Ahookhosh and Amini [An efficient nonmonotone trust-region method for unconstrained optimization, Numer. Algorithms 59 (2012), pp. 523–540]. Reported are numerical results of the proposed algorithm for which the global convergence is established.

A New Restarting Adaptive Trust-Region Method for Unconstrained Optimization

In this paper, we present a new adaptive trust-region method for solving nonlinear unconstrained optimization problems. More precisely, a trust-region radius based on a nonmonotone technique uses an approximation of Hessian which is adaptively chosen. We produce a suitable trust-region radius; preserve the global convergence under classical assumptions to the first-order critical points; improve the practical performance of the new algorithm compared to other exiting variants. Moreover, the quadratic convergence rate is established under suitable conditions. Computational results on the CUTEst test collection of unconstrained problems are presented to show the effectiveness of the proposed algorithm compared with some exiting methods.

Two globally convergent nonmonotone trust-region methods for unconstrained optimization

This paper addresses some trust-region methods equipped with nonmonotone strategies for solving nonlinear unconstrained optimization problems. More specifically, the importance of using nonmonotone techniques in nonlinear optimization is motivated, then two new nonmonotone terms are proposed, and their combinations into the traditional trust-region framework are studied. The global convergence to first- and second-order stationary points and local superlinear and quadratic convergence rates for both algorithms are established. Numerical experiments on the CUTEst test collection of unconstrained problems and some highly nonlinear test functions are reported, where a comparison among state-of-the-art nonmonotone trust-region methods shows the efficiency of the proposed nonmonotone schemes.

A non-monotone retrospective trust-region method for unconstrained optimization

In this paper, a new non-monotone trust-region algorithm is proposed for solving unconstrained nonlinear optimization problems. We modify the retrospective ratio which is introduced by Bastin et al. [Math. Program., Ser. A (2010) 123: 395-418] to form a convex combination ratio for updating the trust-region radius. Then we combine the non-monotone technique with this new framework of trust-region algorithm. The new algorithm is shown to be globally convergent to a first-order criticalpoint. Numerical experiments on CUTEr problems indicate that it is competitive with both the original retrospective trust-region algorithm and the classical trust-region algorithms.

An efficient nonmonotone trust-region method for unconstrained optimization

The monotone trust-region methods are well-known techniques for solving unconstrained optimization problems. While it is known that the nonmonotone strategies not only can improve the likelihood of finding the global optimum but also can improve the numerical performance of approaches, the traditional nonmonotone strategy contains some disadvantages. In order to overcome to these drawbacks, we introduce a variant nonmonotone strategy and incorporate it into trust-region framework to construct more reliable approach. The new nonmonotone strategy is a convex combination of the maximum of function value of some prior successful iterates and the current function value. It is proved that the proposed algorithm possesses global convergence to first-order and second-order stationary points under some classical assumptions. Preliminary numerical experiments indicate that the new approach is considerably promising for solving unconstrained optimization problems.

Global Convergence of General Derivative-Free Trust-Region Algorithms to First- and Second-Order Critical Points

In this paper we prove global convergence for first- and second-order stationary points of a class of derivative-free trust-region methods for unconstrained optimization. These methods are based on the sequential minimization of quadratic (or linear) models built from evaluating the objective function at sample sets. The derivative-free models are required to satisfy Taylor-type bounds, but, apart from that, the analysis is independent of the sampling techniques. A number of new issues are addressed, including global convergence when acceptance of iterates is based on simple decrease of the objective function, trust-region radius maintenance at the criticality step, and global convergence for second-order critical points.

A variant of trust-region methods for unconstrained optimization

In traditional trust-region methods, one, in practice, always employs quadratic model or conic model as the local approximation of the objective function, and there are lots of theoretical results and ripe algorithms. In this paper, we develop a practical trust-region algorithm with a linear model for unconstrained optimization problems. In particular, we combine a special weighted norm with the linear model so that the subproblem contains the information of Hessian matrix of the objective function, which successfully overcome the drawbacks of linear model, and we further complete the trust-region methods with three main types of models, namely, linear model, quadratic model and conic model. We show that the new method preserves the strong global convergence. Moreover, under the linear model, it unveils independently that the line-search algorithms can be viewed as a special case of trust-region methods. Numerical results indicate that the new method is effective and practical.

An adaptive conic trust-region method for unconstrained optimization

It is well known that among the current methods for unconstrained optimization problems the quasi-Newton methods with global strategy may be the most efficient methods, which have local superlinear convergence. However, when the iterative point is far away from the solution of the problem, quasi-Newton method may proceed slowly for the general unconstrained optimization problems. In this article an adaptive conic trust-region method for unconstrained optimization is presented. Not only the gradient information but also the values of the objective function are used to construct the local model at the current iterative point. Moreover, we define a concept of super steepest descent direction and embed its information into the local model. The amount of computation in each iteration of this adaptive algorithm is the same as that of the standard quasi-Newton method with trust region. Some numerical results show that the modified method requires fewer iterations than the standard methods to reach the solution of the optimization problem. Global and local convergence of the method is also analyzed.

Sizing the BFGS and DFP updates: Numerical study

In this study, we develop and test a strategy for selectively sizing (multiplying by an appropriate scalar) the approximate Hessian matrix before it is updated in the BFGS and DFP trust-region methods for unconstrained optimization. Our numerical results imply that, for use with the DFP update, the Oren-Luenberger sizing factor is completely satisfactory and selective sizing is vastly superior to the alternatives of never sizing or first-iteration sizing and is slightly better than the alternative of always sizing. Numerical experimentation showed that the Oren-Luenberger sizing factor is not a satisfactory sizing factor for use with the BFGS update. Therefore, based on our newly acquired understanding of the situation, we propose a centered Oren-Luenberger sizing factor to be used with the BFGS update. Our numerical experimentation implies that selectively sizing the BFGS update with the centered Oren-Luenberger sizing factor is superior to the alternatives. These results contradict the folk axiom that sizing should be done only at the first iteration. They also show that, without sufficient sizing, DFP is vastly inferior to BFGS; however, when selectively sized, DFP is competitive with BFGS.