Learning by maximizing the information transfer through nonlinear noisy neurons and "noise breakdown"

Abstract

The transmission of information through a nonlinear noisy neuron has been computed with the following results. The mutual information between input and output signals is, in the large-noise limit, rigorously given by the mean-squared variance of the fluctuations of the output of the nonlinear neuron. The changes of synaptic strengths that tend to maximize the mutual information are qualitatively similar to those obtained by Hebbian learning of the nonlinear neuron. If noise is added homogeneously to all inputs, its strength becomes multiplied by the number of synapses leading, even in the presence of weak noise, to a ``noise breakdown,'' for which all synaptic strengths tend to the same value during learning.