A stochastic training algorithm for artificial neural networks

Abstract

Random optimization methods typically use Gaussian probability density functions (PDFs) to generate random search vectors. In this paper the random search technique is modified to dynamically seek out the optimal probability density function (OPDF) from which to select the search vector. To this end, the optimization problem is posed as an integral, and the theory of Monte Carlo importance function biasing is applied to determine the theoretical OPDF from which to select changes in the parameters to be optimized. The approach updates its estimate of the OPDF as more information is gained during each training cycle. The dynamic OPDF search process, combined with an adaptive stratified sampling technique, completes the modifications of the basic method. The approach is applied to layered artificial neural networks of generalized, fully interconnected, continuous perceptions and is benchmarked against the backpropagation training method.