New gradient methods with adaptive stepsizes by approximate models

Abstract

As we know, the stepsize is extremely crucial to gradient method. A new type of stepsize is introduced for the gradient method in the paper, which is generated by minimizing the norm of the approximate model of the gradient along the line of negative gradient direction. Based on the retard technique, we present a new gradient method by minimizing adaptively the approximate model of the objective function and the norm of the approximate model of the gradient along the line of negative gradient direction for strictly convex quadratic minimization. The convergence of the proposed method is established. The numerical experiments on four groups of strictly convex quadratic minimization problems illustrate that the proposed method is very promising. We also extend the new gradient method for convex quadratic minimization to general unconstrained optimization by incorporating a nonmonotone line search. The convergence of the resulting method is established. The numerical experiment on the 147 test functions from the CUTEst library indicates that the resulting method is superior to some efficient gradient methods including the BBQ method (SIAM J Optim. 2021;31(4):3068–3096) and is competitive to two famous conjugate gradient software packages CGOPT (1.0) and CGDESCENT (5.0).