Step Replica Symmetry Breaking Research Articles

Replica symmetry breaking in supervised and unsupervised Hebbian networks

Abstract Hebbian neural networks with multi-node interactions, often called Dense Associative Memories, have recently attracted considerable interest in the statistical mechanics community, as they have been shown to outperform their pairwise counterparts in a number of features, including resilience against adversarial attacks, pattern retrieval with extremely weak signals and supra-linear storage capacities. However, their analysis has so far been carried out within a replica-symmetric theory. In this manuscript, we relax the assumption of replica symmetry and analyse these systems at one step of replica-symmetry breaking, focusing on two different prescriptions for the interactions that we will refer to as supervised and unsupervised learning. We derive the phase diagram of the model using two different approaches, namely Parisi's hierarchical ansatz for the relationship between different replicas within the replica approach, and the so-called telescope ansatz within Guerra's interpolation method: our results show that replica-symmetry breaking does not alter the threshold for learning and slightly increases the maximal storage capacity. Further, we also derive analytically the instability line of the replica-symmetric theory, using a generalization of the De Almeida and Thouless approach.

Thermodynamics of bidirectional associative memories

In this paper we investigate the equilibrium properties of bidirectional associative memories (BAMs). Introduced by Kosko in 1988 as a generalization of the Hopfield model to a bipartite structure, the simplest architecture is defined by two layers of neurons, with synaptic connections only between units of different layers: even without internal connections within each layer, information storage and retrieval are still possible through the reverberation of neural activities passing from one layer to another. We characterize the computational capabilities of a stochastic extension of this model in the thermodynamic limit, by applying rigorous techniques from statistical physics. A detailed picture of the phase diagram at the replica symmetric level is provided, both at finite temperature and in the noiseless regimes. Also for the latter, the critical load is further investigated up to one step of replica symmetry breaking. An analytical and numerical inspection of the transition curves (namely critical lines splitting the various modes of operation of the machine) is carried out as the control parameters—noise, load and asymmetry between the two layer sizes—are tuned. In particular, with a finite asymmetry between the two layers, it is shown how the BAM can store information more efficiently than the Hopfield model by requiring less parameters to encode a fixed number of patterns. Comparisons are made with numerical simulations of neural dynamics. Finally, a low-load analysis is carried out to explain the retrieval mechanism in the BAM by analogy with two interacting Hopfield models. A potential equivalence with two coupled Restricted Boltmzann Machines is also discussed.

The discrete random energy model and one step replica symmetry breaking

We solve the random energy model when the energies of the configurations take only integer values. In the thermodynamic limit, the average overlaps remain size dependent and oscillate as the system size increases. While the extensive part of the free energy can still be obtained by a standard replica calculation with one step replica symmetry breaking, it is no longer possible to recover the overlaps in this way. A possible way to adapt the replica approach is to allow the sizes of the blocks in the Parisi matrix to fluctuate and to take complex values.

On Gaussian spin glass with P-wise interactions

The purpose of this paper is to face up the statistical mechanics of dense spin glasses using the well-known Ising case as a prelude for testing the methodologies we develop and then focusing on the Gaussian case as the main subject of our investigation. We tackle the problem of solving for the quenched statistical pressures of these models both at the replica symmetric level and under the first step of replica symmetry breaking by relying upon two techniques: the former is an adaptation of the celebrated Guerra’s interpolation (closer to probability theory in its spirit) and the latter is an adaptation of the transport partial differential equation (closer to mathematical physics in spirit). We recover, in both assumptions, the same expression for quenched statistical pressure and self-consistency equation found with other techniques, including the well-known replica trick technique.

A transport equation approach for deep neural networks with quenched random weights

We consider a multi-layer Sherrington–Kirkpatrick spin-glass as a model for deep restricted Boltzmann machines with quenched random weights and solve for its free energy in the thermodynamic limit by means of Guerra’s interpolating techniques under the RS and 1RSB ansatz. In particular, we recover the expression already known for the replica-symmetric case. Further, we drop the restriction constraint by introducing intra-layer connections among spins and we show that the resulting system can be mapped into a modular Hopfield network, which is also addressed via the same techniques up to the first step of replica symmetry breaking.

One step replica symmetry breaking and overlaps between two temperatures

We obtain an exact analytic expression for the average distribution, in the thermodynamic limit, of overlaps between two copies of the same random energy model (REM) at different temperatures. We quantify the non-self averaging effects and provide an exact approach to the computation of the fluctuations in the distribution of overlaps in the thermodynamic limit. We show that the overlap probabilities satisfy recurrence relations that generalise Ghirlanda–Guerra identities to two temperatures. We also analyse the two temperature REM using the replica method. The replica expressions for the overlap probabilities satisfy the same recurrence relations as the exact form. We show how a generalisation of Parisi’s replica symmetry breaking ansatz is consistent with our replica expressions. A crucial aspect to this generalisation is that we must allow for fluctuations in the replica block sizes even in the thermodynamic limit. This contrasts with the single temperature case where the extremal condition leads to a fixed block size in the thermodynamic limit. Finally, we analyse the fluctuations of the block sizes in our generalised Parisi ansatz and show that in general they may have a negative variance.

Directed Dominating Set Problem Studied by Cavity Method: Warning Propagation and Population Dynamics**Supported by the Doctoral Startup Fund of Xinjiang University of China under Grant No. 208-61357 and the National Natural Science Foundation of China under Grant No. 11765021

The minimal dominating set for a digraph (directed graph) is a prototypical hard combinatorial optimization problem. In a previous paper, we studied this problem using the cavity method. Although we found a solution for a given graph that gives very good estimate of the minimal dominating size, we further developed the one step replica symmetry breaking theory to determine the ground state energy of the undirected minimal dominating set problem. The solution space for the undirected minimal dominating set problem exhibits both condensation transition and cluster transition on regular random graphs. We also developed the zero temperature survey propagation algorithm on undirected Erdős-Rényi graphs to find the ground state energy. In this paper we continue to develope the one step replica symmetry breaking theory to find the ground state energy for the directed minimal dominating set problem. We find the following. (i) The warning propagation equation can not converge when the connectivity is greater than the core percolation threshold value of 3.704. Positive edges have two types warning, but the negative edges have one. (ii) We determine the ground state energy and the transition point of the Erdős-Rényi random graph. (iii) The survey propagation decimation algorithm has good results comparable with the belief propagation decimation algorithm.

Finite Size Corrections to the Parisi Overlap Function in the GREM

We investigate the effects of finite size corrections on the overlap probabilities in the Generalized Random Energy Model (GREM) in two situations where replica symmetry is broken in the thermodynamic limit. Our calculations do not use replicas, but shed some light on what the replica method should give for finite size corrections. In the gradual freezing situation, which is known to exhibit full replica symmetry breaking, we show that the finite size corrections lead to a modification of the simple relations between the sample averages of the overlaps $ Y_k $ between $ k $ configurations predicted by replica theory. This can be interpreted as fluctuations in the replica block size with a \emph{negative} variance. The mechanism is similar to the one we found recently in the random energy model [1]. We also consider a simultaneous freezing situation, which is known to exhibit one step replica symmetry breaking. We show that finite size corrections lead to full replica symmetry breaking and give a more complete derivation of the results presented in [2] for the directed polymer on a tree.

Approximating the XY model on a random graph with aq-state clock model

Numerical simulations of spin glass models with continuous variables set the problem of a reliable but efficient discretization of such variables. In particular, the main question is how fast physical observables computed in the discretized model converge toward the ones of the continuous model when the number of states of the discretized model increases. We answer this question for the XY model and its discretization, the $q$-state clock model, in the mean-field setting provided by random graphs. It is found that the convergence of physical observables is exponentially fast in the number $q$ of states of the clock model, so allowing a very reliable approximation of the XY model by using a rather small number of states. Furthermore, such an exponential convergence is found to be independent from the disorder distribution used. Only at $T=0$ the convergence is slightly slower (stretched exponential). Thanks to the analytical solution to the $q$-state clock model, we compute accurate phase diagrams in the temperature versus disorder strength plane. We find that, at zero temperature, spontaneous replica symmetry breaking takes place for any amount of disorder, even an infinitesimal one. We also study the one step of replica symmetry breaking (1RSB) solution in the low-temperature spin glass phase.

One step replica symmetry breaking and extreme order statistics of logarithmic REMs

Building upon the one-step replica symmetry breaking formalism, we show that the extreme values of a general class of Euclidean-space logarithmic correlated random energy models behave as a randomly shifted decorated exponential Poisson point process in the thermodynamic limit. The distribution of the random shift is determined solely by the large-distance ( “infra-red”, IR) limit of the model, and is equal to the free energy distribution at the critical temperature up to a translation. the decoration process is determined solely by the small-distance (“ultraviolet”, UV) limit, in terms of the biased minimal process. We discuss the relations of our approach with that based on the freezing/duality conjecture, and connections to results in the probability literature. Our approach allowed us to derive the general and explicit formulae for the joint probability density of depths of the first and second minima (as well its higher-order generalizations) in terms of model-specific contributions from UV as well as IR limits. In particular, we show that the distribution of the second minimum is independent of UV data, and depends on IR behaviour via a single parameter, the mean value of the gap. For a given log-correlated field this parameter can be evaluated numerically, and we provide several numerical tests of our theory using the circular model of 1/f-noise.

Finite size corrections in the random energy model and the replica approach

We present a systematic and exact way of computing finite size corrections for the random energy model, in its low temperature phase. We obtain explicit (though complicated) expressions for the finite size corrections of the overlap functions. In its low temperature phase, the random energy model is known to exhibit Parisi's broken symmetry of replicas. The finite size corrections given by our exact calculation can be reproduced using replicas if we make specific assumptions about the fluctuations (with negative variances!) of the number and sizes of the blocks when replica symmetry is broken. As an alternative we show that the exact expression for the non-integer moments of the partition function can be written in terms of coupled contour integrals over what can be thought of as ‘complex replica numbers’. Parisi's one step replica symmetry breaking arises naturally from the saddle point of these integrals without making any ansatz or using the replica method. The fluctuations of the ‘complex replica numbers’ near the saddle point in the imaginary direction correspond to the negative variances we observed in the replica calculation. Finally our approach allows one to see why some apparently diverging series or integrals are harmless.

Spin Glass Field Theory with Replica Fourier Transforms

We develop a field theory for spin glasses using Replica Fourier Transforms (RFT). We present the formalism for the case of replica symmetry and the case of replica symmetry breaking on an ultrametric tree, with the number of replicas n and the number of replica symmetry breaking steps R generic integers. We show how the RFT applied to the two-replica fields allows to construct a new basis which block-diagonalizes the four-replica mass-matrix, into the replicon, anomalous and longitudinal modes. The eigenvalues are given in terms of the mass RFT and the propagators in the RFT space are obtained by inversion of the block-diagonal matrix. The formalism allows to express any i-replica vertex in the new RFT basis and hence enables to perform a standard perturbation expansion. We apply the formalism to calculate the contribution of the Gaussian fluctuations around the Parisi's solution for the free-energy of an Ising spin glass.

Spin glass field theory with replica Fourier transforms

We develop a field theory for spin glasses using replica Fourier transforms (RFT). We present the formalism for the case of replica symmetry and the case of replica symmetry breaking on an ultrametric tree, with the number of replicas n and the number of replica symmetry breaking steps R generic integers. We show how the RFT applied to the two-replica fields allows one to construct a new basis which block-diagonalizes the four-replica mass-matrix, into the replicon, anomalous and longitudinal modes. The eigenvalues are given in terms of the mass RFT and the propagators in the RFT space are obtained by inversion of the block-diagonal matrix. The formalism allows one to express any i-replica vertex in the new RFT basis and hence enables one to perform a standard perturbation expansion. We apply the formalism to calculate the contribution of the Gaussian fluctuations around the Parisi solution for the free-energy of an Ising spin glass.

Condensation in the quantum spherical spin-glass

The quantum spherical spin-glass is analyzed by using a replica symmetric theory (RS) and the physical properties of the system can be explained by the existence of a zero frequency condensate below the critical temperature. The free energy coincides with previous results obtained with an annealed average of the partition function. A calculation with one step replica symmetry breaking (1S-RSB) shows that the only solution is the replica symmetric one.

Replica symmetry breaking, complexity and spin representation in the generalized random energy model

We study the random energy model with a hierarchical structure known as the generalized random energy model (GREM). In contrast to the original analysis by the microcanonical ensemble formalism, we investigate the GREM by the canonical ensemble formalism in conjunction with the replica method. In this analysis, spin-glass-order parameters are defined for the respective hierarchy level, and all possible patterns of replica symmetry breaking (RSB) are taken into account. As a result, we find that the higher step RSB ansatz is useful for describing spin-glass phases in this system. For investigating the nature of the higher step RSB, we generalize the notion of complexity developed for the one-step RSB to the higher step and demonstrate how the GREM is characterized by the generalized complexity. In addition, we propose a novel mean-field spin-glass model with a hierarchical structure, which is equivalent to the GREM at a certain limit. We also show that the same hierarchical structure can be implemented to other mean-field spin models than the GREM. Such models with hierarchy exhibit phase transitions of multiple steps in common.

Stability conditions for fermionic Ising spin-glass models in the presence of a transverse field

The stability of a spin-glass (SG) phase is analyzed in detail for a fermionic Ising SG (FISG) model in the presence of a magnetic transverse field Γ . The fermionic path integral formalism, replica method and static approach have been used to obtain the thermodynamic potential within one step replica symmetry breaking ansatz. The replica symmetry (RS) results show that the SG phase is always unstable against the replicon. Moreover, the two other eigenvalues λ ± of the Hessian matrix (related to the diagonal elements of the replica matrix) can indicate an additional instability to the SG phase, which enhances when Γ is increased. Therefore, this result suggests that the study of the replicon cannot be enough to guarantee the RS stability in the present quantum FISG model, especially near the quantum critical point. In particular, the FISG model allows changing the occupation number of sites, so one can get a first order transition when the chemical potential exceeds a certain value. In this region, the replicon and the λ ± indicate instability problems for the SG solution close to all ranges of a first order boundary.

Finite-size corrections in the Sherrington–Kirkpatrick model

We argue that when the number of spins N in the Sherrington–Kirkpatrick model is finite, the Parisi scheme can be terminated after K replica-symmetry breaking steps, where K(N) ∝ N1/6. We have checked this idea by Monte Carlo simulations: we expect the typical number of peaks and features R in the (non-bond averaged) Parisi overlap function PJ(q) to be of order 2K(N), and our counting (for samples of size N up to 4096 spins) gives results which are consistent with our arguments. We can estimate the leading finite-size correction for any thermodynamic quantity by finding its K-dependence in the Parisi scheme and then replacing K by K(N). Our predictions of how the Edwards–Anderson order parameter and the internal energy of the system approach their thermodynamic limit compare well with the results of our Monte Carlo simulations. The N-dependence of the sample-to-sample fluctuations of thermodynamic quantities can also be obtained; the total internal energy should have sample-to-sample fluctuations of order N1/6, which is again consistent with the results of our numerical simulations.

Small-world hypergraphs on a bond-disordered Bethe lattice

We study the thermodynamic properties of spin systems with bond-disorder on small-world hypergraphs, obtained by superimposing a one-dimensional Ising chain onto a random Bethe graph with p-spin interactions. Using transfer-matrix techniques, we derive fixed-point equations describing the relevant order parameters and the free energy, both in the replica symmetric and one step replica symmetry breaking approximation. We determine the static and dynamic ferromagnetic transition and the spinglass transition within replica symmetry for all temperatures, and demonstrate corrections to these results when one step replica symmetry breaking is taken into account. The results obtained are in agreement with Monte-Carlo simulations.

Stepwise responses in mesoscopic glassy systems: A mean-field approach

We study statistical properties of peculiar responses in glassy systems at mesoscopic scales based on a class of mean-field spin-glass models which exhibit 1 step replica symmetry breaking. Under variation of a generic external field, a finite-sized sample of such a system exhibits a series of step wise responses which can be regarded as a finger print of the sample. We study in detail the statistical properties of the step structures based on a low temperature expansion approach and a replica approach. The spacings between the steps vanish in the thermodynamic limit so that arbitrary small but finite variation of the field induce infinitely many level crossings in the thermodynamic limit leading to a static chaos effect which yields a self-averaging, smooth macroscopic response. We also note that there is a strong analogy between the problem of step-wise responses in glassy systems at mesoscopic scales and intermittency in turbulent flows due to shocks.

Long time limit of equilibrium glassy dynamics and replica calculation

It is shown that the limit t − t ′ → ∞ of the equilibrium dynamic self-energy can be computed from the n → 1 limit of the static self-energy of an n-times replicated system with one step replica symmetry breaking structure. It is also shown that the Dyson equation of the replicated system leads in the n → 1 limit to the bifurcation equation for the glass ergodicity breaking parameter computed from dynamics. The equivalence of the replica formalism to the long time limit of the equilibrium relaxation dynamics is proved to all orders in perturbation for a scalar theory.