Advanced learning methods and exponent regularization applied to a high order neural network

Abstract

High order neural networks (HONNs) are neural networks which employ neurons that combine their inputs non-linearly. The high order network with exponential synaptic links (HONEST) network is a HONN that uses neurons with product units and adaptable exponents. This study examines the use of several advanced learning methods to train the HONEST network: resilient propagation, conjugate gradient, scaled conjugate gradient (SCG), and the Levenberg---Marquardt method. Using a collection of 32 widely-used benchmark datasets, we compare the mean squared error (MSE) performance of the HONEST network across the four algorithms, in addition to backpropagation, and find the SCG method to produce the best performance to a statistically significant extent. Additionally, we investigate the use of a regularization term in the error function, to smooth the magnitudes of the network exponents and nudge the network towards smaller exponents. We find that the use of regularization reduces exponent magnitudes without compromising test set MSE performance.