Techniques of replica symmetry breaking and the storage problem of the McCulloch–Pitts neuron

Abstract

In this article we review the framework for spontaneous replica symmetry breaking. Subsequently that is applied to the example of the statistical mechanical description of the storage properties of a McCulloch–Pitts neuron, i.e., simple perceptron. It is shown that in the neuron problem, the general formula that is at the core of all problems admitting Parisi's replica symmetry breaking ansatz with a one-component order parameter appears. The details of Parisi's method are reviewed extensively, with regard to the wide range of systems where the method may be applied. Parisi's partial differential equation and related differential equations are discussed, and the Green function technique is introduced for the calculation of replica averages, the key to determining the averages of physical quantities. The Green function of the Fokker–Planck equation due to Sompolinsky turns out to play the role of the statistical mechanical Green function in the graph rules for replica correlators. The subsequently obtained graph rules involve only tree graphs, as appropriate for a mean-field-like model. The lowest order Ward–Takahashi identity is recovered analytically and shown to lead to the Goldstone modes in continuous replica symmetry breaking phases. The need for a replica symmetry breaking theory in the storage problem of the neuron has arisen due to the thermodynamical instability of formerly given solutions. Variational forms for the neuron's free energy are derived in terms of the order parameter function x( q), for different prior distribution of synapses. Analytically in the high temperature limit and numerically in generic cases various phases are identified, among them is one similar to the Parisi phase in long-range interaction spin glasses. Extensive quantities like the error per pattern change slightly with respect to the known unstable solutions, but there is a significant difference in the distribution of non-extensive quantities like the synaptic overlaps and the pattern storage stability parameter. A simulation result is also reviewed and compared with the prediction of the theory.