Analysis of Learning from Stochastic Rules in the Framework of Replica Symmetry Breaking

Abstract

We study learning from examples by a perceptron when a return to an example is given by a stochastic relation which is represented by the conditional probability distribution P ( u ), where u is proportional to the inner product between the optimal synaptic weight vector and the example vector. The problem is analyzed by replica method in the case of spherical weights. Since the replica symmetric(RS) solution turns out to be unstable for stochastic cases, we consider the one-step replica symmetry breaking (RSB) solution. We investigate the asymptotic behavior of the learning curve as \(\alpha \equiv \frac{p}{N} \rightarrow \infty\), where p is the number of samples and N is the dimension of the synaptic weights. The average generalization error e g is expressed as (e g - e min ) ∝α - γ in the asymptotic region. For the minimum-error algorithm, we find \(\gamma=\frac{1+\delta}{1+3\delta}\) for the RS solution and \(\gamma=\frac{1+\delta}{1+2\delta}\) for the one-step RSB solution in the case of the functio...