Efficient information transfer and anti-Hebbian neural networks

Abstract

We attempt to maximise mutual information in a neural network where the output signal is corrupted by noise in the transmission process to the next stage. We assume that the average power available for transmission is limited and that the transmission noise is uncorrelated equal-variance Gaussian noise. Using the Lagrange multiplier technique, we construct a utility function that should be maximised to achieve our optimum. This optimum will be achieved when the output signal components are also uncorrelated and of equal variance. For a linear network with lateral inhibitory connections, using an anti-Hebbian algorithm with modified self-inhibitory connections we guarantee to increase our utility function over time and converge to the required optimum, although not by steepest ascent. For a network with inhibitory interneurons, a combined Hebbian/anti-Hebbian algorithm will similarly increase the utility function over time to find the optimum. This network with interneurons suggests that negative feedback may be used in a perceptual system to optimise forward transmission of information.