On the use of simultaneous perturbation stochastic approximation for neural network training

Abstract

Learning, i.e., estimation of weights and biases in neural networks, involves the minimization of a quadratic error criterion, a problem which is usually solved using backpropagation algorithms. This study, which is essentially experimental, aims at assessing the potential of first- and second-order simultaneous perturbation stochastic approximation (SPSA) algorithms to handle this minimization problem. To this end, several application examples in identification and control of nonlinear dynamic systems are presented. Test results, corresponding to training of neural networks possessing different structures and sizes, are discussed in terms of efficiency, accuracy, ease of use (parameter tuning), and implementation.