Abstract
Nesterov accelerated gradient (NAG) method is an efficient first-order algorithm for optimization problems. To ensure the convergence, it usually takes a relatively conservative constant as the stepsize. However, the choice of stepsize has a great impact on the optimization process. In this paper, two adaptive stepsize estimation methods are proposed for complex-valued NAG algorithm for efficient training of fully complex-valued neural networks. The basic idea of the first one is to adaptively determine suitable stepsize by estimating the local smoothness constant of the loss function with the norm of approximate complex Hessian matrix. Its validity is theoretically analyzed by means of the decomposition of complex matrix. Furthermore, by introducing a new parameter design method for multi-step quasi-Newton condition, an improved stepsize estimation is presented. Experimental results on pattern recognition, channel equalization, wind forecasting and synthetic aperture radar (SAR) target classification demonstrate the effectiveness of the proposed methods.
Published Version
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call