Nonmonotone Trust Region Method Research Articles

Effective nonmonotone trust region method based on a simple cubic model for unconstrained optimization problems

Effective nonmonotone trust region method based on a simple cubic model for unconstrained optimization problems

A non-monotone trust-region method with noisy oracles and additional sampling

In this work, we introduce a novel stochastic second-order method, within the framework of a non-monotone trust-region approach, for solving the unconstrained, nonlinear, and non-convex optimization problems arising in the training of deep neural networks. The proposed algorithm makes use of subsampling strategies that yield noisy approximations of the finite sum objective function and its gradient. We introduce an adaptive sample size strategy based on inexpensive additional sampling to control the resulting approximation error. Depending on the estimated progress of the algorithm, this can yield sample size scenarios ranging from mini-batch to full sample functions. We provide convergence analysis for all possible scenarios and show that the proposed method achieves almost sure convergence under standard assumptions for the trust-region framework. We report numerical experiments showing that the proposed algorithm outperforms its state-of-the-art counterpart in deep neural network training for image classification and regression tasks while requiring a significantly smaller number of gradient evaluations.

A novel nonmonotone trust region method based on the Metropolis criterion for solving unconstrained optimization

<p>In this paper, we propose a novel nonmonotone trust region method that incorporates the Metropolis criterion to construct a new function sequence. This sequence is used to update both the trust region ratio and the iteration criterion, increasing the likelihood of accepting the current trial step and introducing randomness into the iteration process. When the current trial step is not accepted, we introduce an improved nonmonotone line search technique to continue the iteration. This approach significantly reduces the number of subproblems that need to be solved, thereby saving computational resources. The stochastic nonmonotone technique helps the algorithm avoid being trapped in the local optima, and a global convergence is guaranteed under certain conditions. Numerical experiments demonstrate that the algorithm can be more effectively applied to a broader range of problems.</p>

A Nonmonotone Trust Region Method for Unconstrained Optimization Problems on Riemannian Manifolds

We propose a nonmonotone trust region method for unconstrained optimization problems on Riemannian manifolds. Global convergence to the first-order stationary points is proved under some reasonable conditions. We also establish local R-linear, super-linear and quadratic convergence rates. Preliminary experiments show that the algorithm is efficient.

An adaptive nonmonotone trust region method based on a modified scalar approximation of the Hessian in the successive quadratic subproblems

Based on a modified secant equation, we propose a scalar approximation of the Hessian to be used in the trust region subproblem. Then, we suggest an adaptive nonmonotone trust region algorithm with a simple quadratic model. Under proper conditions, it is briefly shown that the proposed algorithm is globally and locally superlinearly convergent. Numerical experiments are done on a set of unconstrained optimization test problems of the CUTEr collection, using the Dolan-Moré performance profile. They demonstrate efficiency of the proposed algorithm.

Nonmonotone conic trust region method with line search technique for bound constrained optimization

In this paper, we propose a nonmonotone trust region method for bound constrained optimization problems, where the bounds are dealt with by affine scaling technique. Differing from the traditional trust region methods, the subproblem in our algorithm is based on a conic model. Moreover, when the trial point isn’t acceptable by the usual trust region criterion, a line search technique is used to find an acceptable point. This procedure avoids resolving the trust region subproblem, which may reduce the total computational cost. The global convergence andQ-superlinear convergence of the algorithm are established under some mild conditions. Numerical results on a series of standard test problems are reported to show the effectiveness of the new method.

A non-monotone pattern search approach for systems of nonlinear equations

ABSTRACTIn this paper, a new pattern search is proposed to solve the systems of nonlinear equations. We introduce a new non-monotone strategy which includes a convex combination of the maximum function of some preceding successful iterates and the current function. First, we produce a stronger non-monotone strategy in relation to the generated strategy by Gasparo et al. [Nonmonotone algorithms for pattern search methods, Numer. Algorithms 28 (2001), pp. 171–186] whenever iterates are far away from the optimizer. Second, when iterates are near the optimizer, we produce a weaker non-monotone strategy with respect to the generated strategy by Ahookhosh and Amini [An efficient nonmonotone trust-region method for unconstrained optimization, Numer. Algorithms 59 (2012), pp. 523–540]. Third, whenever iterates are neither near the optimizer nor far away from it, we produce a medium non-monotone strategy which will be laid between the generated strategy by Gasparo et al. [Nonmonotone algorithms for pattern search methods, Numer. Algorithms 28 (2001), pp. 171–186] and Ahookhosh and Amini [An efficient nonmonotone trust-region method for unconstrained optimization, Numer. Algorithms 59 (2012), pp. 523–540]. Reported are numerical results of the proposed algorithm for which the global convergence is established.

A nonmonotone trust-region method for generalized Nash equilibrium and related problems with strong convergence properties

The generalized Nash equilibrium problem (GNEP) is often difficult to solve by Newton-type methods since the problem tends to have locally nonunique solutions. Here we take an existing trust-region method which is known to be locally fast convergent under a relatively mild error bound condition, and modify this method by a nonmonotone strategy in order to obtain a more reliable and efficient solver. The nonmonotone trust-region method inherits the nice local convergence properties of its monotone counterpart and is also shown to have the same global convergence properties. Numerical results indicate that the nonmonotone trust-region method is significantly better than the monotone version, and is at least competitive to an existing software applied to the same reformulation used within our trust-region framework. Additional tests on quasi-variational inequalities (QVI) are also presented to validate efficiency of the proposed extension.

An accelerated nonmonotone trust region method with adaptive trust region for unconstrained optimization

Trust region method is a robust method for optimization problems. In this paper, we propose a novel adaptive nonmonotone technique based on trust region methods for solving unconstrained optimization. In order to accelerate the convergence of trust region methods, an adaptive trust region is generated according to the Hessian of the iterate point. Both the nonmonotone techniques and this adaptive strategies can improve the trust region methods in the sense of convergence. We prove that the proposed method is locally superlinear convergence under some standard assumptions. Numerical results show that the new method is effective and has a high speed of convergence in practice.

A Large-Scale Optimization Method Using a Sparse Approximation of the Hessian for Magnetic Resonance Fingerprinting

Magnetic resonance fingerprinting (MRF) is a novel approach for quantitative imaging which enables the simultaneous determination of multiple tissue-related parameters within short acquisition times. The tissue-related parameters are usually estimated by template matching employing a large dictionary of test signals constructed on the basis of a physical model. We propose to analyze an MRF sequence by a least-squares approach and develop a large-scale optimization algorithm for this purpose. The algorithm is based on a nonmonotone trust region method and utilizes a sparse Jacobian and a sparse approximation of the Hessian. The algorithm is capable of identifying the tissue-related parameters within reasonable calculation times. Simulation results are presented in which the proposed approach compares favorably with previously suggested template matching methods. Moreover, uncertainties for the estimates of the tissue-related parameters calculated on the basis of the approximate Hessian appear to provide a reasonable characterization of their accuracy.

A new class of nonmonotone adaptive trust-region methods for nonlinear equations with box constraints

A nonmonotone trust-region method for the solution of nonlinear systems of equations with box constraints is considered. The method differs from existing trust-region methods both in using a new nonmonotonicity strategy in order to accept the current step and a new updating technique for the trust-region-radius. The overall method is shown to be globally convergent. Moreover, when combined with suitable Newton-type search directions, the method preserves the local fast convergence. Numerical results indicate that the new approach is more effective than existing trust-region algorithms.

A New Nonmonotone Adaptive Retrospective Trust Region Method for Unconstrained Optimization Problems

In this paper, we propose a new nonmonotone adaptive retrospective Trust Region (TR) method for solving unconstrained optimization problems. Inspired by the retrospective ratio proposed in Bastin et al. (Math Program Ser A 123:395---418, 2010), a new nonmonotone TR ratio is introduced based on a convex combination of the nonmonotone classical and retrospective ratios. Due to the value of the new ratio, the TR radius is updated adaptively by a variant of the rule as proposed in Shi and Guo (J Comput Appl Math 213:509---520, 2008). Global convergence property of the new algorithm, as well as its superlinear convergence rate, is established under some standard assumptions. Numerical results on some test problems show the efficiency and effectiveness of the new method in practice, too.

Two globally convergent nonmonotone trust-region methods for unconstrained optimization

This paper addresses some trust-region methods equipped with nonmonotone strategies for solving nonlinear unconstrained optimization problems. More specifically, the importance of using nonmonotone techniques in nonlinear optimization is motivated, then two new nonmonotone terms are proposed, and their combinations into the traditional trust-region framework are studied. The global convergence to first- and second-order stationary points and local superlinear and quadratic convergence rates for both algorithms are established. Numerical experiments on the CUTEst test collection of unconstrained problems and some highly nonlinear test functions are reported, where a comparison among state-of-the-art nonmonotone trust-region methods shows the efficiency of the proposed nonmonotone schemes.

A relaxed nonmonotone adaptive trust region method for solving unconstrained optimization problems

In this paper, we present a new relaxed nonmonotone trust region method with adaptive radius for solving unconstrained optimization problems. The proposed method combines a relaxed nonmonotone technique with a modified version of the adaptive trust region strategy proposed by Shi and Guo (J Comput Appl Math 213:509---520, 2008). Under some suitable and standard assumptions, we establish the global convergence property as well as the superlinear convergence rate for the new method. Numerical results on some test problems show the efficiency and effectiveness of the new proposed method in practice.

Nonmonotone adaptive trust region method with line search based on new diagonal updating

In this paper, a new nonmonotone adaptive trust region method with line search for solving unconstrained nonlinear optimization problems is introduced. The computation of the Hessian approximation is based on the usage of the weak secant equation by a diagonal definite matrix. Under some reasonable conditions, the global convergence of the proposed algorithm is established. The numerical results show the new method is effective and attractive for large scale optimization problems.

A Non-Monotone Trust Region Method with Non-Monotone Wolfe-Type Line Search Strategy for Unconstrained Optimization

In this paper, we propose and analyze a non-monotone trust region method with non-monotone line search strategy for unconstrained optimization problems. Unlike the traditional non-monotone trust region method, our algorithm utilizes non-monotone Wolfe line search to get the next point if a trial step is not adopted. Thus, it can reduce the number of solving sub-problems. Theoretical analysis shows that the new proposed method has a global convergence under some mild conditions.

A nonmonotone trust region method based on simple quadratic models

In this paper, a new nonmonotone trust region algorithm with simple quadratic models is proposed. Unlike traditional nonmonotone trust region method, our trust region subproblem is very simple by using a new scale approximation of the minimizing function’s Hessian. The global convergence of the proposed algorithm is established under some reasonable conditions. Numerical tests on a set of large scale standard test problems are presented and show that the new algorithm is efficient and robust.

Nonmonotone adaptive trust region method based on simple conic model for unconstrained optimization

We propose a nonmonotone adaptive trust region method based on simple conic model for unconstrained optimization. Unlike traditional trust region methods, the subproblem in our method is a simple conic model, where the Hessian of the objective function is approximated by a scalar matrix. The trust region radius is adjusted with a new self-adaptive adjustment strategy which makes use of the information of the previous iteration and current iteration. The new method needs less memory and computational efforts. The global convergence and Q-superlinear convergence of the algorithm are established under the mild conditions. Numerical results on a series of standard test problems are reported to show that the new method is effective and attractive for large scale unconstrained optimization problems.

A Nonmonotone Adaptive Trust Region Method Based on Conic Model for Unconstrained Optimization

We propose a nonmonotone adaptive trust region method for unconstrained optimization problems which combines a conic model and a new update rule for adjusting the trust region radius. Unlike the traditional adaptive trust region methods, the subproblem of the new method is the conic minimization subproblem. Moreover, at each iteration, we use the last and the current iterative information to define a suitable initial trust region radius. The global and superlinear convergence properties of the proposed method are established under reasonable conditions. Numerical results show that the new method is efficient and attractive for unconstrained optimization problems.

A new trust region method for solving least-square transformation of system of equalities and inequalities

In this paper, a new nonmonotone trust region method with adaptive radius is proposed for solving system of equalities and inequalities. This method combines a new nonmonotone technique with a new adaptive strategy based on the Shi and Guo’s adaptive technique in (J Comput Appl Math 213:509–520, 2008), which makes full use of the current point information. Under some standard assumptions, the global convergence property as well as the superlinear convergence rate are established for the new method. Numerical results on some nonlinear systems of equalities and inequalities indicate the efficiency and robustness of the proposed method in practice.