Stochastic adaptive CL-BFGS algorithms for fully complex-valued dendritic neuron model

Abstract

This paper proposes two stochastic variance reduced gradient algorithms based on adaptive complex-valued limited-memory BFGS (ACL-BFGS) algorithm, which are established on the calculation of Wirtinger gradient for fully complex-valued dendritic neuron model (FCDNM). Basically, the explicit expression of Wirtinger gradient is derived to enable the quasi-Newton algorithm for the training of FCDNM. A stochastic variance reduction strategy is adopted such that the number of samples per batch is increased during the optimization process to ensure the effectiveness of the stochastic algorithm. The stochastic ACL-BFGS algorithm with variance reduction (SACL-BFGS) can be efficiently applied for the training of FCDNM. Furthermore, by generalizing the A3 accumulative multi-step quasi-Newton method, the A3-SACL-BFGS algorithm is presented to achieve fast convergence. The experimental results on some benchmark prediction and classification problems conclusively imply that the proposed SACL-BFGS and A3-SACL-BFGS algorithms for FCDNM are clearly superior to complex-valued backpropagation (CBP), CL-BFGS and ACL-BFGS algorithms.