Adaptive Nonmonotone Research Articles

A nonmonotone adaptive trust region technique with a forgetting factor

Nonmonotonicity has been considered to be essentially influential on the efficiency of the iterative procedures of nonlinear optimization. A review of the literature shows that the objective function values available from recent iterations provide worthier information in the nonmonotone schemes. So, with the aim of enhancing probability of applying more recent available function values, a nonmonotone trust region ratio is suggested using a forgetting factor. Meanwhile, modification of a recent adaptive formula for the trust region radius is devised by a nonmonotone reflection as well. Then, based on the two mentioned modifications, an adaptive nonmonotone trust region algorithm is given. In addition, convergence of the method is analysed under classic assumptions. To provide support for our theoretical arguments, computational merits of the given algorithm on a set of CUTEr test functions are depicted.

Nonmonotone variable metric Barzilai-Borwein method for composite minimization problem

<abstract><p>In this study, we develop a nonmonotone variable metric Barzilai-Borwein method for minimizing the sum of a smooth function and a convex, possibly nondifferentiable, function. At each step, the descent direction is obtained by taking the difference between the minimizer of the scaling proximal function and the current iteration point. An adaptive nonmonotone line search is proposed for determining the step length along this direction. We also show that the limit point of the iterates sequence is a stationary point. Numerical results with parallel magnetic resonance imaging, Poisson, and Cauchy noise deblurring demonstrate the effectiveness of the new algorithm.</p></abstract>

Two adaptive nonmonotone trust-region algorithms for solving multiobjective optimization problems

ABSTRACT This paper attempts to propose two adaptive nonmonotone trust-region algorithms in the multiobjective optimization (MO) case. The first proposed trust-region algorithm uses a modified maximum nonmonotone technique, which takes a convex combination of the maximum value of some preceding successful iterates and the function value in the current iterate. The second one employs the average nonmonotone technique, which takes a weighted average of the successive function values. Under some suitable assumptions, the convergence of the sequences generated by the trust-region algorithms that use the aforementioned nonmonotone techniques to a critical point is shown. Using some well-known test problems, we compare our proposed adaptive nonmonotone MO algorithms with some other MO trust-region algorithms. To conduct a thorough comparison in this regard, some performance criteria are used. These numerical results confirm a significant advantage of applying the proposed adaptive nonmonotone trust-region algorithms in solving MO problems. Finally, the proposed algorithms are implemented to optimize one of the optimization problems of the abrasive water-jet machining process.

Proximal gradient methods beyond monotony

We address composite optimization problems, which consist in minimizing the sum of a smooth and a merely lower semicontinuous function, without any convexity assumptions. Numerical solutions of these problems can be obtained by proximal gradient methods, which often rely on a line search procedure as globalization mechanism. We consider an adaptive nonmonotone proximal gradient scheme based on an averaged merit function and establish asymptotic convergence guarantees under weak assumptions, delivering results on par with the monotone strategy. Global worst-case rates for the iterates and a stationarity measure are also derived. Finally, a numerical example indicates the potential of nonmonotonicity and spectral approximations.

An Adaptive Nonmonotone Line Search Technique for Solving Systems of Nonlinear Equations

In this article, a new nonmonotone line search technique is proposed for solving a system of nonlinear equations. We attempt to answer this question how to control the degree of the nonmonotonicity of line search rules in order to reach a more efficient algorithm? Therefore, we present a novel algorithm that can avoid the increase of unsuccessful iterations. For this purpose, we show the robust behavior of the proposed algorithm by solving a few numerical examples. Under some suitable assumptions, the global convergence of our strategy is proved.

An adaptive nonmonotone line search for multiobjective optimization problems

This paper aims to propose an adaptive nonmonotone line search for direction-based multiobjective optimization (MO) algorithms. All direction-based MO algorithms can be equipped with this line search, especially the BFGS quasi-Newton algorithm is applied in this study. To validate the proposed line search some well-known line searches, including Armijo, maximum nonmonotone and average nonmonotone line searches, were considered. In order to make a comprehensive comparison between the proposed line sereach and the aforementioned line searches two criteria are considered: the computational effort and quality of the approximated nondominated frontier. The results confirm the remarkable superiority of the proposed line search over the mentioned line searches. In addition, this line search preserves the quality of the obtained nondominated frontier. It should be noted that we established the convergency of BFGS quasi-Newton algorithm equipped with the proposed line search under some mild conditions.

An Adaptive Nonmonotone Projected Barzilai-Borwein Gradient Method with Active Set Prediction for Nonnegative Matrix Factorization

An Adaptive Nonmonotone Projected Barzilai-Borwein Gradient Method with Active Set Prediction for Nonnegative Matrix Factorization

Gradient methods exploiting spectral properties

We propose a new stepsize for the gradient method. It is shown that this new stepsize will converge to the reciprocal of the largest eigenvalue of the Hessian, when Dai-Yang's asymptotic optimal gradient method (Computational Optimization and Applications, 2006, 33(1): 73–88) is applied for minimizing quadratic objective functions. Based on this spectral property, we develop a monotone gradient method that takes a certain number of steps using the asymptotically optimal stepsize by Dai and Yang, and then follows by some short steps associated with this new stepsize. By employing one step retard of the asymptotic optimal stepsize, a nonmonotone variant of this method is also proposed. Under mild conditions, R-linear convergence of the proposed methods is established for minimizing quadratic functions. In addition, by combining gradient projection techniques and adaptive nonmonotone line search, we further extend those methods for general bound constrained optimization. Two variants of gradient projection methods combining with the Barzilai-Borwein stepsizes are also proposed. Our numerical experiments on both quadratic and bound constrained optimization indicate that the new proposed strategies and methods are very effective.

An Efficient Nonmonotone Method for State-Constrained Elliptic Optimal Control Problems

This paper presents a novel numerical strategy based on combination of an adaptive semismooth Newton (ASN) method and the Lavrentiev regularization technique for the solution of elliptic optimal control problems with state constraints. Using the global convergence proof for a nonmonotone semismooth Newton method, we will exploit an adaptive nonmonotone line search method such that the nonmonotonicity degree of this method can be increased when the results are far from the optimum solution and it can be reduced when they are close to the optimizer. In this strategy, the role of the Lavrentiev regularization technique is converting the original optimal control problem to a regularized optimal control problem. Using the finite difference discretization scheme and a Newton–Cotes rule, the regularized optimal control problem is converted to a bound constrained optimization problem (BCOP). Then the ASN method is implemented to solve the resulting BCOP. Numerical results show the efficiency of the proposed procedure.

An adaptive nonmonotone trust region method based on a modified scalar approximation of the Hessian in the successive quadratic subproblems

Based on a modified secant equation, we propose a scalar approximation of the Hessian to be used in the trust region subproblem. Then, we suggest an adaptive nonmonotone trust region algorithm with a simple quadratic model. Under proper conditions, it is briefly shown that the proposed algorithm is globally and locally superlinearly convergent. Numerical experiments are done on a set of unconstrained optimization test problems of the CUTEr collection, using the Dolan-Moré performance profile. They demonstrate efficiency of the proposed algorithm.

An adaptive nonmonotone active set – weight constrained – neural network training algorithm

In this work, a new direction for improving the classification accuracy of artificial neural networks is proposed by bounding the weights of the network, during the training process. Furthermore, a new adaptive nonmonotone active set – weight constrained – neural network training algorithm is proposed in order to demonstrate the efficacy and efficiency of our approach. The proposed training algorithm consists of two phases: a gradient projection phase which utilizes an adaptive nonmonotone line search and an unconstrained optimization phase which exploits the box structure of the bounds. Also, a set of switching criteria is defined for efficiently switching between the two phases. Our preliminary numerical experiments illustrate that the classification efficiency of the proposed algorithm outperforms classical neural network training algorithms, providing empirical evidence that it provides more stable, efficient and reliable learning.

An adaptive nonmonotone truncated Newton method for optimal control of a class of parabolic distributed parameter systems

A black-box method using the finite elements, the Crank–Nicolson and a nonmonotone truncated Newton (TN) method is presented for solving optimal control problems (OCPs) governed by partial differential equations (PDEs). The proposed method finds the optimal control of a class of linear and nonlinear parabolic distributed parameter systems with a quadratic cost functional. To this end, the piecewise linear finite elements method and the well-known Crank–Nicolson method are used for discretizing in space and in time, respectively. Afterwards, regarding the implicit function theorem (IFT), the optimal control problem is transformed into an unconstrained nonlinear optimization problem. Considering that in a gradient-based method for solving optimal control problems, the evaluations of gradients and Hessians of the cost functional is important, hence, an adjoint technique is used to evaluate them effectively. In addition, to make a globalization strategy, we first introduce an adaptive nonmonotone strategy which properly controls the degree of nonmonotonicity and then incorporate it into an inexact Armijo-type line search approach to construct a more relaxed line search procedure. Finally, the obtained unconstrained nonlinear optimization problem is solved by utilizing the proposed nonmonotone truncated Newton method. Results gained from the new offered method compared with existing methods show that the new method is promising.

An adaptive semismooth Newton method for approximately solving control-constrained elliptic optimal control problems

The main goal of this paper is to study, analyse and prove the global convergence property of a nonmonotone semismooth Newton method. Based on this property, we aim to exploit an adaptive nonmonotone line search method and apply it for the approximate solution of elliptic optimal control problems with control constraints. To this end, we first convert the optimal control problem to a bound constrained optimization problem by using a finite-difference discretization scheme and a Newton–Cotes rule. Then we implement the adaptive nonmonotone semismooth Newton method to the resulting bound constrained optimization problem. We give some numerical examples to demonstrate the validity, efficiency and applicability of the presented method.

An accelerated nonmonotone trust region method with adaptive trust region for unconstrained optimization

Trust region method is a robust method for optimization problems. In this paper, we propose a novel adaptive nonmonotone technique based on trust region methods for solving unconstrained optimization. In order to accelerate the convergence of trust region methods, an adaptive trust region is generated according to the Hessian of the iterate point. Both the nonmonotone techniques and this adaptive strategies can improve the trust region methods in the sense of convergence. We prove that the proposed method is locally superlinear convergence under some standard assumptions. Numerical results show that the new method is effective and has a high speed of convergence in practice.

An adaptive nonmonotone trust region algorithm

Based on an eigenvalue analysis conducted on the scaled memoryless quasi-Newton updating formulas BFGS and DFP, an adaptive choice for the trust region radius is proposed. Then, using a trust region ratio obtained from a nonmonotone line search strategy, an adaptive nonmonotone trust region algorithm is developed. Under proper conditions, it is briefly shown that the proposed algorithm is globally and locally superlinearly convergent. Numerical experiments are done on a set of unconstrained optimization test problems of the CUTEr collection, using the Dolan–Moré performance profile. They show efficiency of the proposed algorithm.

Optimal control of viscous Burgers equation via an adaptive nonmonotone Barzilai–Borwein gradient method

ABSTRACTAn adaptive nonmonotone spectral gradient method for the solution of distributed optimal control problem (OCP) for the viscous Burgers equation is presented in a black-box framework. Regarding the implicit function theorem, the OCP is transformed into an unconstrained nonlinear optimization problem (UNOP). For solving UNOP, an adaptive nonmonotone Barzilai–Borwein gradient method is proposed in which to make a globalization strategy, first an adaptive nonmonotone strategy which properly controls the degree of nonmonotonicity is presented and then is incorporated into an inexact line search approach to construct a more relaxed line search procedure. Also an adjoint technique is used to effectively evaluate the gradient. The low memory requirement and the guaranteed convergence property make the proposed method quite useful for large-scale OCPs. The efficiency of the presented method is supported by numerical experiments.

An adaptive nonmonotone global Barzilai–Borwein gradient method for unconstrained optimization

The Barzilai–Borwein (BB) gradient method has received many studies due to its simplicity and numerical efficiency. By incorporating a nonmonotone line search, Raydan (SIAM J Optim. 1997;7:26–33) has successfully extended the BB gradient method for solving general unconstrained optimization problems so that it is competitive with conjugate gradient methods. However, the numerical results reported by Raydan are poor for very ill-conditioned problems because the effect of the degree of nonmonotonicity may be noticeable. In this paper, we focus more on the nonmonotone line search technique used in the global Barzilai–Borwein (GBB) gradient method. We improve the performance of the GBB gradient method by proposing an adaptive nonmonotone line search based on the morphology of the objective function. We also prove the global convergence and the R-linear convergence rate of the proposed method under reasonable assumptions. Finally, we give some numerical experiments made on a set of unconstrained optimization test problems of the CUTEr collection. The results show the efficiency of the proposed method in the sense of the performance profile introduced (Math Program. 2002;91:201–213) by Dolan and Moré.

An Efficient Augmented Lagrangian Method for Statistical X-Ray CT Image Reconstruction.

Statistical iterative reconstruction (SIR) for X-ray computed tomography (CT) under the penalized weighted least-squares criteria can yield significant gains over conventional analytical reconstruction from the noisy measurement. However, due to the nonlinear expression of the objective function, most exiting algorithms related to the SIR unavoidably suffer from heavy computation load and slow convergence rate, especially when an edge-preserving or sparsity-based penalty or regularization is incorporated. In this work, to address abovementioned issues of the general algorithms related to the SIR, we propose an adaptive nonmonotone alternating direction algorithm in the framework of augmented Lagrangian multiplier method, which is termed as “ALM-ANAD”. The algorithm effectively combines an alternating direction technique with an adaptive nonmonotone line search to minimize the augmented Lagrangian function at each iteration. To evaluate the present ALM-ANAD algorithm, both qualitative and quantitative studies were conducted by using digital and physical phantoms. Experimental results show that the present ALM-ANAD algorithm can achieve noticeable gains over the classical nonlinear conjugate gradient algorithm and state-of-the-art split Bregman algorithm in terms of noise reduction, contrast-to-noise ratio, convergence rate, and universal quality index metrics.

Barzilai-Borwein-like methods for the extreme eigenvalue problem

We consider numerical methods for the extreme eigenvalue problem of large scale symmetric positive definite matrices. By the variational principle, the extreme eigenvalue can be obtained by minimizing some unconstrained optimization problem. Firstly, we propose two adaptive nonmonotone Barzilai-Borwein-like methods for the unconstrained optimization problem. Secondly, we prove the global convergence of the two algorithms under some conditions. Thirdly, we compare our methods with eigsand the power method for the standard test problems from the UF Sparse Matrix Collection. The primary numerical experiments indicate that the two algorithms are promising.

A framework of constraint preserving update schemes for optimization on Stiefel manifold

This paper considers optimization problems on the Stiefel manifold $X^{\mathsf{T}}X=I_p$, where $X\in \mathbb{R}^{n \times p}$ is the variable and $I_p$ is the $p$-by-$p$ identity matrix. A framework of constraint preserving update schemes is proposed by decomposing each feasible point into the range space of $X$ and the null space of $X^{\mathsf{T}}$. While this general framework can unify many existing schemes, a new update scheme with low complexity cost is also discovered. Then we study a feasible Barzilai-Borwein-like method under the new update scheme. The global convergence of the method is established with an adaptive nonmonotone line search. The numerical tests on the nearest low-rank correlation matrix problem, the Kohn-Sham total energy minimization and a specific problem from statistics demonstrate the efficiency of the new method. In particular, the new method performs remarkably well for the nearest low-rank correlation matrix problem in terms of speed and solution quality and is considerably competitive with the widely used SCF iteration for the Kohn-Sham total energy minimization.