Replica-symmetric Mean-field Theory Research Articles

Capacity of the covariance perceptron

The classical perceptron is a simple neural network that performs a binary classification by a linear mapping between static inputs and outputs and application of a threshold. For small inputs, neural networks in a stationary state also perform an effectively linear input–output transformation, but of an entire time series. Choosing the temporal mean of the time series as the feature for classification, the linear transformation of the network with subsequent thresholding is equivalent to the classical perceptron. Here we show that choosing covariances of time series as the feature for classification maps the neural network to what we call a ‘covariance perceptron’; a mapping between covariances that is bilinear in terms of weights. By extending Gardner’s theory of connections to this bilinear problem, using a replica symmetric mean-field theory, we compute the pattern and information capacities of the covariance perceptron in the infinite-size limit. Closed-form expressions reveal superior pattern capacity in the binary classification task compared to the classical perceptron in the case of a high-dimensional input and low-dimensional output. For less convergent networks, the mean perceptron classifies a larger number of stimuli. However, since covariances span a much larger input and output space than means, the amount of stored information in the covariance perceptron exceeds the classical counterpart. For strongly convergent connectivity it is superior by a factor equal to the number of input neurons. Theoretical calculations are validated numerically for finite size systems using a gradient-based optimization of a soft-margin, as well as numerical solvers for the NP hard quadratically constrained quadratic programming problem, to which training can be mapped.

Optimal Segmentation of Directed Graph and the Minimum Number of Feedback Arcs

The minimum feedback arc set problem asks to delete a minimum number of arcs (directed edges) from a digraph (directed graph) to make it free of any directed cycles. In this work we approach this fundamental cycle-constrained optimization problem by considering a generalized task of dividing the digraph into D layers of equal size. We solve the D-segmentation problem by the replica-symmetric mean field theory and belief-propagation heuristic algorithms. The minimum feedback arc density of a given random digraph ensemble is then obtained by extrapolating the theoretical results to the limit of large D. A divide-and-conquer algorithm (nested-BPR) is devised to solve the minimum feedback arc set problem with very good performance and high efficiency.

Spin-glass phase transitions and minimum energy of the random feedback vertex set problem.

A feedback vertex set (FVS) of an undirected graph contains vertices from every cycle of this graph. Constructing a FVS of sufficiently small cardinality is very difficult in the worst cases, but for random graphs this problem can be efficiently solved by converting it into an appropriate spin-glass model [H.-J. Zhou, Eur. Phys. J. B 86, 455 (2013)EPJBFY1434-602810.1140/epjb/e2013-40690-1]. In the present work we study the spin-glass phase transitions and the minimum energy density of the random FVS problem by the first-step replica-symmetry-breaking (1RSB) mean-field theory. For both regular random graphs and Erdös-Rényi graphs, we determine the inverse temperature β_{l} at which the replica-symmetric mean-field theory loses its local stability, the inverse temperature β_{d} of the dynamical (clustering) phase transition, and the inverse temperature β_{s} of the static (condensation) phase transition. These critical inverse temperatures all change with the mean vertex degree in a nonmonotonic way, and β_{d} is distinct from β_{s} for regular random graphs of vertex degrees K>60, while β_{d} are identical to β_{s} for Erdös-Rényi graphs at least up to mean vertex degree c=512. We then derive the zero-temperature limit of the 1RSB theory and use it to compute the minimum FVS cardinality.

Statistical mechanics of the fashion game on random networks

A model of fashion on networks is studied. This model consists of two groups of agents that are located on a network and have opposite viewpoints towards being fashionable: behaving consistently with either the majority or the minority of adjacent agents. Checking whether the fashion game has a pure Nash equilibrium (pure NE) is a non-deterministic polynomial complete problem. Using replica-symmetric mean field theory, the largest proportion of satisfied agents and the region where at least one pure NE should exist are determined for several types of random networks. Furthermore, a quantitive analysis of the asynchronous best response dynamics yields the phase diagram of existence and detectability of pure NE in the fashion game on some random networks.

Retrieval behavior and thermodynamic properties of symmetrically diluted Q-Ising neural networks.

The retrieval behavior and thermodynamic properties of symmetrically diluted Q-Ising neural networks are derived and studied in replica-symmetric mean-field theory generalizing earlier works on either the fully connected or the symmetrical extremely diluted network. Capacity-gain parameter phase diagrams are obtained for the Q=3, Q=4, and Q=infinity state networks with uniformly distributed patterns of low activity in order to search for the effects of a gradual dilution of the synapses. It is shown that enlarged regions of continuous changeover into a region of optimal performance are obtained for finite stochastic noise and small but finite connectivity. The de Almeida-Thouless lines of stability are obtained for arbitrary connectivity, and the resulting phase diagrams are used to draw conclusions on the behavior of symmetrically diluted networks with other pattern distributions of either high or low activity.

Magnetic order and categorization in attractor neural networks

The thermodynamic behavior of an extended Hopfield model for an attractor neural network with units in three Ising-states S i and symmetrically random diluted synaptic connections J ij d, between units i and j, is discussed here in the thermodynamic limit within replica-symmetric mean-field theory. A macroscopic set of random patterns in a two-level hierarchical structure of ancestors and descendents is used to construct the synaptic connections and to determine the magnetic order associated with the recognition of ancestors.

Thermodynamic properties of extremely diluted symmetricQ-Ising neural networks

Using the replica-symmetric mean-field theory approach the thermodynamic and retrieval properties of extremely diluted {\it symmetric} $Q$-Ising neural networks are studied. In particular, capacity-gain parameter and capacity-temperature phase diagrams are derived for $Q=3, 4$ and $Q=\infty$. The zero-temperature results are compared with those obtained from a study of the dynamics of the model. Furthermore, the de Almeida-Thouless line is determined. Where appropriate, the difference with other $Q$-Ising architectures is outlined.

Categorization in fully connected multistate neural network models.

The categorization ability of fully connected neural network models, with either discrete or continuous Q-state units, is studied in this work in replica symmetric mean-field theory. Hierarchically correlated multistate patterns in a two level structure of ancestors and descendents (examples) are embedded in the network and the categorization task consists in recognizing the ancestors when the network is trained exclusively with their descendents. Explicit results for the dependence of the equilibrium properties of a Q=3-state model and a Q=infinity-state model are obtained in the form of phase diagrams and categorization curves. A strong improvement of the categorization ability is found when the network is trained with examples of low activity. The categorization ability is found to be robust to finite threshold and synaptic noise. The Almeida-Thouless lines that limit the validity of the replica-symmetric results, are also obtained.

Categorization in the symmetrically dilute Hopfield network.

A symmetrically dilute Hopfield model with a Hebbian learning rule is used to study the effects of gradual dilution and of synaptic noise on the categorization ability of an attractor neural network with hierarchically correlated patterns in a two-level structure of ancestors and descendants. Categorization consists in recognizing the ancestors when the network has been trained exclusively with the descendants. We consider a macroscopic number of ancestors, each with a finite number of descendants, and take into account the stochastic noise produced by the former in an equilibrium study of the network, by means of replica-symmetric mean-field theory. Phase diagrams are obtained that exhibit a categorization, a spin-glass, and a paramagnetic phase, as well as the dependence of the order parameters on the relevant quantities. The de Almeida-Thouless lines that limit the validity of the replica-symmetric results are also obtained. It is shown that gradual dilution increases considerably the region where a stable categorization phase may be found.

Ising spin glass: Replica-symmetric cluster expansion in finite dimensions.

The replica-symmetric cluster expansion for the Ising spin glass is reviewed and explained. The expansion allows computation of various quantities and correlations directly in finite dimensions, without recourse to the mean-field theory or the saddle point. The expansion for the free energy is recovered with the use of external fields that couple to the replica-symmetric Edwards-Anderson order parameter. The expansion scheme is applied to the Edwards-Anderson susceptibility. The expansion diverges in the mean-field [infinite dimensionality] limit. The divergent expansion, when naively summed, yields the negative replicon mass of the mean-field theory. This so-called Almeida-Thouless instability of the replica-symmetric mean-field theory is argued to be absent in finite dimensions. The argument, similar to the one already reported for the entropy, depends on the low-temperature behavior of terms in the expansion at finite dimensions. \textcopyright{} 1996 The American Physical Society.

On the multi-neuron interaction model without truncating the interaction

A replica-symmetric mean-field theory approach is presented to the multi-neuron interaction model introduced by de Almeida and Iglesias (1990 Phys. Lett. 146A 239). Fixed-point equations are derived for the relevant order parameters of the model, extended to include biased patterns, without truncating the interaction. The capacity-bias and the temperature-capacity phase diagrams are discussed. Compared with the truncated version of the model, it is found that the capacity at zero temperature is infinite and that the retrieval states satisfy the de Almeida-Thouless stability condition.

Neural networks with low levels of activity: Ising vs. McCulloch-Pitts neurons

The performance of neural networks used as associative memory for uncorrelated patterns with prescribed mean activity is analyzed within the replica symmetric mean field theory. The optimal representation of the possible states of the neutrons, active or inactive, is found to depend on the mean activity. For activity equal one half Ising neurons and for low activities McCulloch-Pitts neurons are optimal. In this optimal representation the noise due to noncondensed patterns is reduced.

Partially connected models of neural networks

A partially connected Hopfield neural network model is studied under the restriction that w, the ratio of connections per site to the size of the system, remains finite as the size N to infinity with the connection structure at each site being the same. The replica symmetric mean field theory equations for the order parameters are derived. The zero-temperature forms of these equations are then solved numerically for a few different 'local' connectivity architectures showing phase transitions at different critical storage ratios, alpha c, where the states which the authors are trying to store in the network become discontinuously unstable. They show that the information capacity per connection improves for partially connected systems.

Random-field smearing of the proton-glass transition.

Within a replica-symmetric mean-field theory we have studied the proton pseudo-spin-glass behavior of a random-bond classical Ising system in a homogeneous transverse field \ensuremath{\Omega} and a random longitudinal field. This model is expected to describe some properties of the mixed hydrogen-bonded ferro- and antiferroelectric crystals such as ${\mathrm{Rb}}_{1\mathrm{\ensuremath{-}}\mathit{x}}$(${\mathrm{NH}}_{4}$${)}_{\mathit{x}}$${\mathrm{H}}_{2}$${\mathrm{PO}}_{4}$ which have recently been investigated experimentally. It is shown that in the presence of Gaussian random fields with zero mean and variance \ensuremath{\Delta} the proton-glass transition is smeared out, i.e., the cusp in the dielectric susceptibility is rounded off and the proton-glass order parameter remains finite at temperatures above the nominal freezing temperature. However, the average dielectric polarization is strictly zero for a symmetric bond distribution. We have also determined the limits of stability of the replica-symmetric solution for the case of a deuterated system (\ensuremath{\Omega}=0). The replica-symmetric proton-glass phase is separated from the phase with broken replica symmetry by a line of instability in the (T,\ensuremath{\Delta}) plane. The crossing of this line is thus connected with a phase transition which persists in the presence of random fields. Finally, the distribution function of local parallel fields P(h) determining the magnetic resonance line shape has been calculated within the random-field model and the results applied to interpret some recent experimental data.

Magnetic behavior of the Co2−xZnxTiO4 and Co2SnO4 systems

Our ac susceptibility (χ) measurements, carried out between 1.7 and 100 K, show five χ peaks, signaling five transition temperatures, in the disordered frustrated spinel ferrite Co2TiO4. This result, in conjunction with the replica symmetric mean field theory of vector spin glasses and earlier magnetization and neutron diffraction studies, indicates a separate freezing of A- and B-site spins. To our knowledge, such a phenomenon has not been observed before. Co2SnO4 and Co1.2Zn0.8TiO4 χ measurements, where three and one χ peaks are respectively observed, support this picture.

Information storage in neural networks with low levels of activity.

The Hopfield model of a neural network is extended to allow for the storage and retrieval of biased patterns, {${\ensuremath{\xi}}_{i}^{\ensuremath{\mu}}$}, where ${N}^{\mathrm{\ensuremath{-}}1}$${\mathcal{J}}_{i}^{N}$${\ensuremath{\xi}}_{i}^{\ensuremath{\mu}}$=a is arbitrary. Such patterns represent levels of activity (i.e., percentage of firing neurons) equal to (1/2)(1+a), -1<a<1. If the coupling constants (synaptic efficacies) are constructed as in the original Hopfield model, the system can retrieve at most a very small number of patterns (p1+${a}^{\mathrm{\ensuremath{-}}2}$). This is due to the finite correlations (overlaps) between the patterns. The model is modified by subtracting the bias a from each pattern as it enters into the couplings. This modification restores the ability of the model to store a macroscopic number of patterns. Yet spurious states are found to plague the dynamics. It is then argued that the dynamics of the network should be consistent with the levels of activity of the stored patterns. This is implemented by adding a global constraint, which restricts the configuration space to states whose mean activity is in the neighborhood of (1/2)(1+a). The consequences of the restricted dynamics are analyzed in the replica symmetric mean-field theory. The global constraint suppresses spurious states and leads to the unexpected result that the storage capacity is higher than that of the unbiased network, up to very high values of the bias (\ensuremath{\Vert}a\ensuremath{\Vert}\ensuremath{\simeq}0.99). However, the information content of such networks is shown to be a monotonically decreasing function of a.