Semi-local units for prediction

Abstract

The authors consider a class of semilocal activation functions, which respond to more localized regions of input space than sigmoid functions but less localized regions than radial basis functions (RBFs). In particular, they examine Gaussian bar functions, which sum the Gaussian responses from each input dimension. They present evidence that Gaussian bar networks avoid the slow learning problems of sigmoid networks and deal more robustly with irrelevant inputs than RBF networks. On the Mackey-Glass problem, the speedup over sigmoid networks is so dramatic that the difference in training time between RBF and Gaussian bar networks is minor. Architectures that superpose composed Gaussians (Gaussians-of Gaussians) to approximate the unknown function have the best performance. An automatic connection pruning mechanism inherent in the Gaussian bar function is very likely a key factor in the success of this representation. >