On lateral connections in feed-forward neural networks

Abstract

Feed-forward neural networks trained with back-propagation, or a variation of it, constitute the most widely applied of all synthetic neural network paradigms. Several efforts have been directed towards faster training of such networks. Most of these efforts attempt to take as large steps as possible during the training process. However, another potential source of inefficiency arising from the independent evolution of the hidden layer neurons has received considerably less attention. Fahlman and Lebiere (1991) have called this the herd effect. The herd effect arises because all the hidden layer neurons receive similar information, and thus evolve to reduce the then largest source of error during the training process. In this paper we introduce local lateral connections in a feed-forward network towards reducing this effect. A new algorithm, based on gradient descent, is derived for the proposed architecture and its efficacy evaluated through simulations.