Nonmonotone Technique Research Articles

An Improved Non-Monotonic Adaptive Trust Region Algorithm for Unconstrained Optimization

The trust region method is an effective method for solving unconstrained optimization problems. Incorrectly updating the rules of the trust region radius will increase the number of iterations and affect the calculation efficiency. In order to obtain an effective radius for the trust region, an adaptive radius updating criterion is proposed based on the gradient of the current iteration point and the eigenvalue of the Hessian matrix which avoids calculating the inverse of the Hessian matrix during radius updating. This approach reduces the computation time and enhances the algorithm’s performance. On this basis, we apply adaptive radius and non-monotonic techniques to the trust region algorithm and propose an improved non-monotonic adaptive trust region algorithm. Under proper assumptions, the convergence of the algorithm is analyzed. Numerical experiments confirm that the suggested algorithm is effective.

A New Adaptive Levenberg–Marquardt Method for Nonlinear Equations and Its Convergence Rate under the Hölderian Local Error Bound Condition

The Levenberg–Marquardt (LM) method is one of the most significant methods for solving nonlinear equations as well as symmetric and asymmetric linear equations. To improve the method, this paper proposes a new adaptive LM algorithm by modifying the LM parameter, combining the trust region technique and the non-monotone technique. It is interesting that the new algorithm is constantly optimized by adaptively choosing the LM parameter. To evaluate the effectiveness of the new algorithm, we conduct tests using various examples. To extend the convergence results, we prove the convergence of the new algorithm under the Hölderian local error bound condition rather than the commonly used local error bound condition. Theoretical analysis and numerical results show that the new algorithm is stable and effective.

A novel nonmonotone trust region method based on the Metropolis criterion for solving unconstrained optimization

<p>In this paper, we propose a novel nonmonotone trust region method that incorporates the Metropolis criterion to construct a new function sequence. This sequence is used to update both the trust region ratio and the iteration criterion, increasing the likelihood of accepting the current trial step and introducing randomness into the iteration process. When the current trial step is not accepted, we introduce an improved nonmonotone line search technique to continue the iteration. This approach significantly reduces the number of subproblems that need to be solved, thereby saving computational resources. The stochastic nonmonotone technique helps the algorithm avoid being trapped in the local optima, and a global convergence is guaranteed under certain conditions. Numerical experiments demonstrate that the algorithm can be more effectively applied to a broader range of problems.</p>

Two adaptive nonmonotone trust-region algorithms for solving multiobjective optimization problems

ABSTRACT This paper attempts to propose two adaptive nonmonotone trust-region algorithms in the multiobjective optimization (MO) case. The first proposed trust-region algorithm uses a modified maximum nonmonotone technique, which takes a convex combination of the maximum value of some preceding successful iterates and the function value in the current iterate. The second one employs the average nonmonotone technique, which takes a weighted average of the successive function values. Under some suitable assumptions, the convergence of the sequences generated by the trust-region algorithms that use the aforementioned nonmonotone techniques to a critical point is shown. Using some well-known test problems, we compare our proposed adaptive nonmonotone MO algorithms with some other MO trust-region algorithms. To conduct a thorough comparison in this regard, some performance criteria are used. These numerical results confirm a significant advantage of applying the proposed adaptive nonmonotone trust-region algorithms in solving MO problems. Finally, the proposed algorithms are implemented to optimize one of the optimization problems of the abrasive water-jet machining process.

An accelerated alternating directional method with non-monotone technique for matrix recovery

<abstract><p>Low-rank and sparse structures have been frequently exploited in matrix recovery and robust PCA problems. In this paper, we develop an alternating directional method and its variant equipped with the non-monotone search procedure for solving a non-convex optimization model of low-rank and sparse matrix recovery problems, where the concerned matrix with incomplete data is separable into a low-rank part and a sparse part. The main idea is to use the alternating minimization method for the low-rank matrix part, and to use the non-monotone line search technique for the sparse matrix part to iteratively update, respectively. To some extent, the non-monotone strategy relaxes the single-step descent into a multi-step descent and then greatly improves the performance of the alternating directional method. Theoretically, we prove the global convergence of the two proposed algorithms under some mild conditions. Finally, the comparison of numerical experiments shows that the alternate directional method with non-monotone technique is more effective than the original monotone method and the previous method. The efficiency and effectiveness of the proposed algorithms are demonstrated by solving some instances of random incomplete matrix recovery problems and some problems of the background modeling in video processing.</p></abstract>

A regularized limited memory BFGS method for large-scale unconstrained optimization and its efficient implementations

The limited memory BFGS (L-BFGS) method is one of the popular methods for solving large-scale unconstrained optimization. Since the standard L-BFGS method uses a line search to guarantee its global convergence, it sometimes requires a large number of function evaluations. To overcome the difficulty, we propose a new L-BFGS with a certain regularization technique. We show its global convergence under the usual assumptions. In order to make the method more robust and efficient, we also extend it with several techniques such as the nonmonotone technique and simultaneous use of the Wolfe line search. Finally, we present some numerical results for test problems in CUTEst, which show that the proposed method is robust in terms of solving more problems.

A new limited memory method for unconstrained nonlinear least squares

This paper suggests a new limited memory trust region algorithm for large unconstrained black box least squares problems, called LMLS. Main features of LMLS are a new non-monotone technique, a new adaptive radius strategy, a new Broyden-like algorithm based on the previous good points, and a heuristic estimation for the Jacobian matrix in a subspace with random basis indices. Our numerical results show that LMLS is robust and efficient, especially in comparison with solvers using traditional limited memory and standard quasi-Newton approximations.

A Spectral Gradient Projection Method for the Positive Semi-definite Procrustes Problem

This paper addresses the positive semi-deffnite procrustes problem (PSDP). The PSDP corresponds to a least squares problem over the set of symmetric and semi-deffnite positive matrices. These kinds of problems appear in many applications such as structure analysis, signal processing, among others. A non-monotone spectral projected gradient algorithm is proposed to obtain a numerical solution for the PSDP. The proposed algorithm employs the Zhang and Hager's non-monotone technique in combination with the Barzilai and Borwein's step size to accelerate convergence. Some theoretical results are presented. Finally, numerical experiments are performed to demonstrate the effectiveness and efficiency of the proposed method, and comparisons are made with other state-of-the-art algorithms.

A New Modified Efficient Levenberg–Marquardt Method for Solving Systems of Nonlinear Equations

For systems of nonlinear equations, a modified efficient Levenberg–Marquardt method with new LM parameters was developed by Amini et al. (2018). The convergence of the method was proved under the local error bound condition. In order to enhance this method, using nonmonotone technique, we propose a new Levenberg–Marquardt parameter in this paper. The convergence of the new Levenberg–Marquardt method is shown to be at least superlinear, and numerical experiments show that the new Levenberg–Marquardt algorithm can solve systems of nonlinear equations effectively.

A nonmonotone accelerated Levenberg–Marquardt method for the ‐eigenvalues of symmetric tensors

AbstractThe eigenvalues of tensors become more and more important in the numerical multilinear algebra. In this paper, based on the nonmonotone technique, an accelerated Levenberg–Marquardt (LM) algorithm is presented for computing the ‐eigenvalues of symmetric tensors, in which an LM step and an accelerated LM step are computed at each iteration. We establish the global convergence of the proposed algorithm using properties of symmetric tensors and norms. Under the local error‐bound condition, the cubic convergence of the nonmonotone accelerated LM algorithm is derived. Numerical results show that this method is efficient.

A New Nonmonotone Adaptive Trust Region Method

The trust region method plays an important role in solving optimization problems. In this paper, we propose a new nonmonotone adaptive trust region method for solving unconstrained optimization problems. Actually, we combine a popular nonmonotone technique with an adaptive trust region algorithm. The new ratio to adjusting the next trust region radius is different from the ratio in the traditional trust region methods. Under some appropriate conditions, we show that the new algorithm has good global convergence and superlinear convergence.

Using a trust region method with nonmonotone technique to solve unrestricted optimization problem

In this work, we combine adaptive trust region method TRM with nonmonotone strategy to introduce a new algorithm to solve systems of unconstrained optimization problems depending on the simple quadratic model. The subproblems will be solving easier with the new approximation of the Hessian function. This technique does not need large capacity and is characterized by ease of application. We established the algorithm convergence with standard assumptions. The numerical results compered the performance of the new approach with three famous algorithms in terms of the time consumed to find the solution in addition to the number of iterations and the number of functions evaluations. From these results we can show that the new method is the best among the traditional algorithms.

Nonmonotone line searches for unconstrained multiobjective optimization problems

In the last two decades, many descent methods for multiobjective optimization problems were proposed. In particular, the steepest descent and the Newton methods were studied for the unconstrained case. In both methods, the search directions are computed by solving convex subproblems, and the stepsizes are obtained by an Armijo-type line search. As a consequence, the objective function values decrease at each iteration of the algorithms. In this work, we consider nonmonotone line searches, i.e., we allow the increase of objective function values in some iterations. Two well-known types of nonmonotone line searches are considered here: the one that takes the maximum of recent function values, and the one that takes their average. We also propose a new nonmonotone technique specifically for multiobjective problems. Under reasonable assumptions, we prove that every accumulation point of the sequence produced by the nonmonotone version of the steepest descent and Newton methods is Pareto critical. Moreover, we present some numerical experiments, showing that the nonmonotone technique is also efficient in the multiobjective case.

A penalty-free method with superlinear convergence for equality constrained optimization

In this paper, we propose a new penalty-free method for solving nonlinear equality constrained optimization. This method uses different trust regions to cope with the nonlinearity of the objective function and the constraints instead of using a penalty function or a filter. To avoid Maratos effect, we do not make use of the second order correction or the nonmonotone technique, but utilize the value of the Lagrangian function instead of the objective function in the acceptance criterion of the trial step. The feasibility restoration phase is not necessary, which is often used in filter methods or some other penalty-free methods. Global and superlinear convergence are established for the method under standard assumptions. Preliminary numerical results are reported, which demonstrate the usefulness of the proposed method.

English

Trust region methods are a class of effective iterative schemes in numerical optimization. In this paper, a new improved nonmonotone adaptive trust region method for solving unconstrained optimization problems is proposed. We construct an approximate model where the approximation to Hessian matrix is updated by the scaled memoryless BFGS update formula, and incorporate a nonmonotone technique with the new proposed adaptive trust region radius. The new ratio to adjusting the next trust region radius is different from the ratio in the traditional trust region methods. Under some suitable and standard assumptions, it is shown that the proposed algorithm possesses global convergence and superlinear convergence. Numerical results demonstrate that the proposed method is very promising.

A new nonmonotone line-search trust-region approach for nonlinear systems

This paper introduces a new derivative-free trust-region algorithm for solving nonlinear systems, based on a new nonmonotone technique and an adaptive radius strategy. It is shown that we can generate the small (large) steps and radii in the cases where iterations are near (far away from) the optimizer. Such a nonmonotone strategy is embedded into the trust region framework and Armijo line search to face with problems which have the narrow curved valley. To prevent resolving the trust-region subproblem, the nonmonotone Armijo line search is used whenever iterations are unsuccessful. In each iteration, the adaptive radius strategy is constructed based on the norm of the best function values. The global and q-quadratic rate of convergence of the new algorithm is proved. Computational results are reported.

A Shamanskii-like self-adaptive Levenberg–Marquardt method for nonlinear equations

In this paper, we are concerned with the Shamanskii-like self-adaptive Levenberg–Marquardt methods for nonlinear equations. We consider two choices of Levenberg–Marquardt parameter. One of them is the standard self-adaptive Levenberg–Marquardt parameter, the other is nonmonotone self-adaptive Levenberg–Marquardt parameter by using the nonmonotone technique of Grippo, Lampariello and Lucidi. Under the error bound condition which is weaker than nonsingularity, we show that the Shamanskii-like self-adaptive Levenberg–Marquardt methods converge with Q-order (m+1). Numerical experiments show the new algorithms are efficient and promising.

On the convergence rate of scaled gradient projection method

The scaled gradient projection (SGP) method, which can be viewed as a promising improvement of the classical gradient projection method, is a quite efficient solver for real-world problems arising in image science and machine learning. Most recently, Bonettini and Prato [Inverse Probl. 2015;31:095008. 20 p] proved that the SGP method with the monotone Armijo line search technique has the convergence rate, where k counts the iteration. In this paper, we first show that the SGP method could be equipped with the nonmonotone line search procedure proposed by Zhang and Hager [SIAM J Optim. 2004;14:1043–1056]. To some extent, such a nonmonotone technique might improve the performance of SGP method, because its effectiveness has been verified for unconstrained optimization by comparing with the traditional monotone and nonmonotone strategies. Then, we prove that the new SGP method also has the convergence rate under the condition that the objective function is convex. Furthermore, we derive the linear convergence of the SGP algorithm under the strongly convexity assumption of the involved objective function.

An efficient Levenberg–Marquardt method with a new LM parameter for systems of nonlinear equations

A modified Levenberg–Marquardt method for solving singular systems of nonlinear equations was proposed by Fan [J Comput Appl Math. 2003;21;625–636]. Using trust region techniques, the global and quadratic convergence of the method were proved. In this paper, to improve this method, we decide to introduce a new Levenberg–Marquardt parameter while also incorporate a new nonmonotone technique to this method. The global and quadratic convergence of the new method is proved under the local error bound condition. Numerical results show the new algorithm is efficient and promising.

A Nonmonotone Trust Region Method for Nonsmooth Composite Programming Problems

A class of nonsmooth composite minimization problems are considered in this paper. In practice, many problems in computational science, engineering and other enormous areas fall into this class of problems. Two obvious applications of this class of problems are least square problems with nonsmooth data and the problem of solving a system of nonsmooth equations. Because of its practical importance, this class of problem has received wide attention from the mathematical optimization community. In particular, Sampaio, Yuan and Sun has proposed a trust region method for this class of problems and also provided the convergence analysis to support their algorithm. Motivated partly by their work, in this paper a nonmonotone trust region algorithm for this class of nonsmooth composite minimization problems is presented. Different from most existing monotone line search and trust region methods, this method combines the nonmonotone technique to improve the efficiency of the trust region method. After a brief introduction of the class of problems in the first section, some fundamental concepts and properties which will be used in this paper are presented. Then, the new nonmonotone trust region algorithm for the class of problem is described followed by the global convergence analysis of the new algorithm. A simple application of this algorithm is discussed in the last part of this paper.