RasID - Random Search for Neural Network Training

Abstract

This paper presents a novel random search, RasID, for neural network training, that introduces a sophisticated probability density function (PDF) in a random search for generating search vectors. The PDF provides two parameters to control local search ranges and directions efficiently. This realizes an intensified search where it is easy to find good solutions locally or a diversified search to escape local minima based on success-failure of past searches. Local gradients, if available, and trend information on the criterion function surface are used to improve search performance. The proposed scheme is applied to layered neural network training and is benchmarked against deterministic and nondeterministic methods.