Adaptive Nonmonotone Line Search Research Articles

Nonmonotone variable metric Barzilai-Borwein method for composite minimization problem

<abstract><p>In this study, we develop a nonmonotone variable metric Barzilai-Borwein method for minimizing the sum of a smooth function and a convex, possibly nondifferentiable, function. At each step, the descent direction is obtained by taking the difference between the minimizer of the scaling proximal function and the current iteration point. An adaptive nonmonotone line search is proposed for determining the step length along this direction. We also show that the limit point of the iterates sequence is a stationary point. Numerical results with parallel magnetic resonance imaging, Poisson, and Cauchy noise deblurring demonstrate the effectiveness of the new algorithm.</p></abstract>

An Adaptive Nonmonotone Line Search Technique for Solving Systems of Nonlinear Equations

In this article, a new nonmonotone line search technique is proposed for solving a system of nonlinear equations. We attempt to answer this question how to control the degree of the nonmonotonicity of line search rules in order to reach a more efficient algorithm? Therefore, we present a novel algorithm that can avoid the increase of unsuccessful iterations. For this purpose, we show the robust behavior of the proposed algorithm by solving a few numerical examples. Under some suitable assumptions, the global convergence of our strategy is proved.

An adaptive nonmonotone line search for multiobjective optimization problems

This paper aims to propose an adaptive nonmonotone line search for direction-based multiobjective optimization (MO) algorithms. All direction-based MO algorithms can be equipped with this line search, especially the BFGS quasi-Newton algorithm is applied in this study. To validate the proposed line search some well-known line searches, including Armijo, maximum nonmonotone and average nonmonotone line searches, were considered. In order to make a comprehensive comparison between the proposed line sereach and the aforementioned line searches two criteria are considered: the computational effort and quality of the approximated nondominated frontier. The results confirm the remarkable superiority of the proposed line search over the mentioned line searches. In addition, this line search preserves the quality of the obtained nondominated frontier. It should be noted that we established the convergency of BFGS quasi-Newton algorithm equipped with the proposed line search under some mild conditions.

Gradient methods exploiting spectral properties

We propose a new stepsize for the gradient method. It is shown that this new stepsize will converge to the reciprocal of the largest eigenvalue of the Hessian, when Dai-Yang's asymptotic optimal gradient method (Computational Optimization and Applications, 2006, 33(1): 73–88) is applied for minimizing quadratic objective functions. Based on this spectral property, we develop a monotone gradient method that takes a certain number of steps using the asymptotically optimal stepsize by Dai and Yang, and then follows by some short steps associated with this new stepsize. By employing one step retard of the asymptotic optimal stepsize, a nonmonotone variant of this method is also proposed. Under mild conditions, R-linear convergence of the proposed methods is established for minimizing quadratic functions. In addition, by combining gradient projection techniques and adaptive nonmonotone line search, we further extend those methods for general bound constrained optimization. Two variants of gradient projection methods combining with the Barzilai-Borwein stepsizes are also proposed. Our numerical experiments on both quadratic and bound constrained optimization indicate that the new proposed strategies and methods are very effective.

An Efficient Nonmonotone Method for State-Constrained Elliptic Optimal Control Problems

This paper presents a novel numerical strategy based on combination of an adaptive semismooth Newton (ASN) method and the Lavrentiev regularization technique for the solution of elliptic optimal control problems with state constraints. Using the global convergence proof for a nonmonotone semismooth Newton method, we will exploit an adaptive nonmonotone line search method such that the nonmonotonicity degree of this method can be increased when the results are far from the optimum solution and it can be reduced when they are close to the optimizer. In this strategy, the role of the Lavrentiev regularization technique is converting the original optimal control problem to a regularized optimal control problem. Using the finite difference discretization scheme and a Newton–Cotes rule, the regularized optimal control problem is converted to a bound constrained optimization problem (BCOP). Then the ASN method is implemented to solve the resulting BCOP. Numerical results show the efficiency of the proposed procedure.

An adaptive nonmonotone active set – weight constrained – neural network training algorithm

In this work, a new direction for improving the classification accuracy of artificial neural networks is proposed by bounding the weights of the network, during the training process. Furthermore, a new adaptive nonmonotone active set – weight constrained – neural network training algorithm is proposed in order to demonstrate the efficacy and efficiency of our approach. The proposed training algorithm consists of two phases: a gradient projection phase which utilizes an adaptive nonmonotone line search and an unconstrained optimization phase which exploits the box structure of the bounds. Also, a set of switching criteria is defined for efficiently switching between the two phases. Our preliminary numerical experiments illustrate that the classification efficiency of the proposed algorithm outperforms classical neural network training algorithms, providing empirical evidence that it provides more stable, efficient and reliable learning.

An adaptive semismooth Newton method for approximately solving control-constrained elliptic optimal control problems

The main goal of this paper is to study, analyse and prove the global convergence property of a nonmonotone semismooth Newton method. Based on this property, we aim to exploit an adaptive nonmonotone line search method and apply it for the approximate solution of elliptic optimal control problems with control constraints. To this end, we first convert the optimal control problem to a bound constrained optimization problem by using a finite-difference discretization scheme and a Newton–Cotes rule. Then we implement the adaptive nonmonotone semismooth Newton method to the resulting bound constrained optimization problem. We give some numerical examples to demonstrate the validity, efficiency and applicability of the presented method.

An adaptive nonmonotone global Barzilai–Borwein gradient method for unconstrained optimization

The Barzilai–Borwein (BB) gradient method has received many studies due to its simplicity and numerical efficiency. By incorporating a nonmonotone line search, Raydan (SIAM J Optim. 1997;7:26–33) has successfully extended the BB gradient method for solving general unconstrained optimization problems so that it is competitive with conjugate gradient methods. However, the numerical results reported by Raydan are poor for very ill-conditioned problems because the effect of the degree of nonmonotonicity may be noticeable. In this paper, we focus more on the nonmonotone line search technique used in the global Barzilai–Borwein (GBB) gradient method. We improve the performance of the GBB gradient method by proposing an adaptive nonmonotone line search based on the morphology of the objective function. We also prove the global convergence and the R-linear convergence rate of the proposed method under reasonable assumptions. Finally, we give some numerical experiments made on a set of unconstrained optimization test problems of the CUTEr collection. The results show the efficiency of the proposed method in the sense of the performance profile introduced (Math Program. 2002;91:201–213) by Dolan and Moré.

An Efficient Augmented Lagrangian Method for Statistical X-Ray CT Image Reconstruction.

Statistical iterative reconstruction (SIR) for X-ray computed tomography (CT) under the penalized weighted least-squares criteria can yield significant gains over conventional analytical reconstruction from the noisy measurement. However, due to the nonlinear expression of the objective function, most exiting algorithms related to the SIR unavoidably suffer from heavy computation load and slow convergence rate, especially when an edge-preserving or sparsity-based penalty or regularization is incorporated. In this work, to address abovementioned issues of the general algorithms related to the SIR, we propose an adaptive nonmonotone alternating direction algorithm in the framework of augmented Lagrangian multiplier method, which is termed as “ALM-ANAD”. The algorithm effectively combines an alternating direction technique with an adaptive nonmonotone line search to minimize the augmented Lagrangian function at each iteration. To evaluate the present ALM-ANAD algorithm, both qualitative and quantitative studies were conducted by using digital and physical phantoms. Experimental results show that the present ALM-ANAD algorithm can achieve noticeable gains over the classical nonlinear conjugate gradient algorithm and state-of-the-art split Bregman algorithm in terms of noise reduction, contrast-to-noise ratio, convergence rate, and universal quality index metrics.

On the rate of convergence of projected Barzilai–Borwein methods

We study the rate of convergence of a projected Barzilai–Borwein method, which performs the Grippo–Lampariello–Lucidi (GLL) non-monotone line search along the feasible direction, for convex constrained optimization. Under mild conditions, we establish sublinear convergence of the considered method for general convex functions. Moreover, we show that the rate of convergence is R-linear for strongly convex functions. We extend the convergence results to different versions of the method based on a general GLL non-monotone line search, an adaptive non-monotone line search, and a non-monotone line search that requires an average of the successive function values decreases.

A framework of constraint preserving update schemes for optimization on Stiefel manifold

This paper considers optimization problems on the Stiefel manifold $X^{\mathsf{T}}X=I_p$, where $X\in \mathbb{R}^{n \times p}$ is the variable and $I_p$ is the $p$-by-$p$ identity matrix. A framework of constraint preserving update schemes is proposed by decomposing each feasible point into the range space of $X$ and the null space of $X^{\mathsf{T}}$. While this general framework can unify many existing schemes, a new update scheme with low complexity cost is also discovered. Then we study a feasible Barzilai-Borwein-like method under the new update scheme. The global convergence of the method is established with an adaptive nonmonotone line search. The numerical tests on the nearest low-rank correlation matrix problem, the Kohn-Sham total energy minimization and a specific problem from statistics demonstrate the efficiency of the new method. In particular, the new method performs remarkably well for the nearest low-rank correlation matrix problem in terms of speed and solution quality and is considerably competitive with the widely used SCF iteration for the Kohn-Sham total energy minimization.

Feasible Barzilai–Borwein-like methods for extreme symmetric eigenvalue problems

This paper aims to study feasible Barzilai–Borwein (BB)-like methods for extreme symmetric eigenvalue problems. For the two-dimensional case, we establish the local superlinear convergence result of FLBB, FSBB, FABB, and FADBB algorithms. A counter-example is also given, showing that the algorithms may cycle or stop at a non-stationary point. In order to circumvent the difficulty, we propose a safeguard in choosing the stepsize. We also adopt an adaptive non-monotone line search with an improved line search to ensure the global convergence of AFBB-like methods. Numerical experiments on a set of test problems from UF Sparse Matrix Collection demonstrate that, comparing several available codes including eigs, irbleigs and jdcg, AFBB-like methods are very useful for large-scale sparse extreme symmetric eigenvalue problems.

A nonmonotone spectral projected gradient method for large-scale topology optimization problems

An efficient gradient-based method to solve the volume constrained topology optimization problems is presented. Each iterate of this algorithm is obtained by the projection of a Barzilai-Borwein step onto the feasible set consisting of box and one linear constraints (volume constraint). To ensure the global convergence, an adaptive nonmonotone line search is performed along the direction that is given by the current and projection point. The adaptive cyclic reuse of the Barzilai-Borwein step is applied as the initial stepsize. The minimum memory requirement, the guaranteed convergence property, and almost only one function and gradient evaluations per iteration make this new method very attractive within common alternative methods to solve large-scale optimal design problems. Efficiency and feasibility of the presented method are supported by numerical experiments.

On Nonmonotone Chambolle Gradient Projection Algorithms for Total Variation Image Restoration

The main aim of this paper is to accelerate the Chambolle gradient projection method for total variation image restoration. In the proposed minimization method model, we use the well known Barzilai-Borwein stepsize instead of the constant time stepsize in Chambolle's method. Further, we adopt the adaptive nonmonotone line search scheme proposed by Dai and Fletcher to guarantee the global convergence of the proposed method. Numerical results illustrate the efficiency of this method and indicate that such a nonmonotone method is more suitable to solve some large-scale inverse problems.

Adaptive nonmonotone line search method for unconstrained optimization

In this paper, an adaptive nonmonotone line search method for unconstrained minimization problems is proposed. At every iteration, the new algorithm selects only one of the two directions: a Newton-type direction and a negative curvature direction, to perform the line search. The nonmonotone technique is included in the backtracking line search when the Newton-type direction is the search direction. Furthermore, if the negative curvature direction is the search direction, we increase the steplength under certain conditions. The global convergence to a stationary point with second-order optimality conditions is established. Some numerical results which show the efficiency of the new algorithm are reported.

New algorithms for singly linearly constrained quadratic programs subject to lower and upper bounds

There are many applications related to singly linearly constrained quadratic programs subjected to upper and lower bounds. In this paper, a new algorithm based on secant approximation is provided for the case in which the Hessian matrix is diagonal and positive definite. To deal with the general case where the Hessian is not diagonal, a new efficient projected gradient algorithm is proposed. The basic features of the projected gradient algorithm are: 1) a new formula is used for the stepsize; 2) a recently-established adaptive non-monotone line search is incorporated; and 3) the optimal stepsize is determined by quadratic interpolation if the non-monotone line search criterion fails to be satisfied. Numerical experiments on large-scale random test problems and some medium-scale quadratic programs arising in the training of Support Vector Machines demonstrate the usefulness of these algorithms.

Adaptive Two-Point Stepsize Gradient Algorithm

Combined with the nonmonotone line search, the two-point stepsize gradient method has successfully been applied for large-scale unconstrained optimization. However, the numerical performances of the algorithm heavily depend on M, one of the parameters in the nonmonotone line search, even for ill-conditioned problems. This paper proposes an adaptive nonmonotone line search. The two-point stepsize gradient method is shown to be globally convergent with this adaptive nonmonotone line search. Numerical results show that the adaptive nonmonotone line search is specially suitable for the two-point stepsize gradient method.