Replica Symmetric Research Articles

Counting and hardness-of-finding fixed points in cellular automata on random graphs

Abstract We study the fixed points of outer-totalistic cellular automata on sparse random regular graphs. These can be seen as constraint satisfaction problems, where each variable must adhere to the same local constraint, which depends solely on its state and the total number of its neighbors in each possible state. Examples of this setting include classical problems such as independent sets or assortative/dissasortative partitions. We analyse the existence and number of fixed points in the large system limit using the cavity method, under both the replica symmetric (RS) and one-step replica symmetry breaking (1RSB) assumption. This method allows us to characterize the structure of the space of solutions, in particular, if the solutions are clustered and whether the clusters contain frozen variables. This last property is conjectured to be linked to the typical algorithmic hardness of the problem. We bring experimental evidence for this claim by studying the performance of the belief-propagation reinforcement algorithm, a message-passing-based solver for these constraint satisfaction problems.

Effect of constraint relaxation on the minimum vertex cover problem in random graphs.

A statistical-mechanical study of the effect of constraint relaxation on the minimum vertex cover problem in Erdős-Rényi random graphs is presented. Using a penalty-method formulation for constraint relaxation, typical properties of solutions, including infeasible solutions that violate the constraints, are analyzed by means of the replica method and cavity method. The problem involves a competition between reducing the number of vertices to be covered and satisfying the edge constraints. The analysis under the replica-symmetric (RS) ansatz clarifies that the competition leads to degeneracies in the vertex and edge states, which determine the quantitative properties of the system, such as the cover and penalty ratios. A precise analysis of these effects improves the accuracy of RS approximation for the minimum cover ratio in the replica symmetry-breaking (RSB) region. Furthermore, the analysis based on the RS cavity method indicates that the RS/RSB boundary of the ground states with respect to the mean degree of the graphs is expanded, and the critical temperature is lowered by constraint relaxation.

Replica analysis of overfitting in regression models for time to event data: the impact of censoring

We use statistical mechanics techniques, viz. the replica method, to model the effect of censoring on overfitting in Cox’s proportional hazards model, the dominant regression method for time-to-event data. In the overfitting regime, Maximum Likelihood (ML) parameter estimators are known to be biased already for small values of the ratio of the number of covariates over the number of samples. The inclusion of censoring was avoided in previous overfitting analyses for mathematical convenience, but is vital to make any theory applicable to real-world medical data, where censoring is ubiquitous. Upon constructing efficient algorithms for solving the new (and more complex) Replica Symmetric (RS) equations and comparing the solutions with numerical simulation data, we find excellent agreement, even for large censoring rates. We then address the practical problem of using the theory to correct the biased ML estimators without knowledge of the data-generating distribution. This is achieved via a novel numerical algorithm that self-consistently approximates all relevant parameters of the data generating distribution while simultaneously solving the RS equations. We investigate numerically the statistics of the corrected estimators, and show that the proposed new algorithm indeed succeeds in removing the bias of the ML estimators, for both the association parameters and for the cumulative hazard.

Ising spin glass on random graphs at zero temperature: Not all spins are glassy in the glassy phase

We investigate the replica symmetry broken (RSB) phase of spin glass (SG) models in a random field defined on Bethe lattices at zero temperature. From the properties of the RSB solution we deduce a closed equation for the extreme values of the cavity fields. This equation turns out not to depend on the parameters defining the RSB, and it predicts that the spontaneous RSB does not take place homogeneously on the whole system. Indeed, there exist spins having the same effective local field in all local ground states, exactly as in the replica symmetric (RS) phase, while the spontaneous RSB manifests only on the remaining spins, whose fraction vanishes at criticality. The characterization in terms of spins having fixed or fluctuating local fields can be extended also to the random field Ising model (RFIM), in which case the fluctuating spins are the only responsible for the spontaneous magnetization in the ferromagnetic phase. Close to criticality we are able to connect the statistics of the local fields acting on the spins in the RSB phase with the correlation functions measured in the paramagnetic phase. Identifying the two types of spins on given instances of SG and RFIM, we show that they participate very differently to avalanches produced by flipping a single spin. From the scaling of the number of spins inducing RSB effects close to the critical point and using the $M$-layer expansion we estimate the upper critical dimension $D_U \geq 8$ for SG.

Planted XY model: Thermodynamics and inference.

In this paper we study a fully connected planted spin glass named the planted XY model. Motivation for studying this system comes both from the spin glass field and the one of statistical inference where it models the angular synchronization problem. We derive the replica symmetric (RS) phase diagram in the temperature, ferromagnetic bias plane using the approximate message passing (AMP) algorithm and its state evolution (SE). While the RS predictions are exact on the Nishimori line (i.e., when the temperature is matched to the ferromagnetic bias), they become inaccurate when the parameters are mismatched, giving rise to a spin glass phase where AMP is not able to converge. To overcome the defects of the RS approximation we carry out a one-step replica symmetry-breaking (1RSB) analysis based on the approximate survey propagation (ASP) algorithm. Exploiting the state evolution of ASP, we count the number of metastable states in the measure, derive the 1RSB free entropy and find the behavior of the Parisi parameter throughout the spin glass phase.

Ising spin glass in a random network with a Gaussian random field.

We investigate thermodynamic phase transitions of the joint presence of spin glass (SG) and random field (RF) using a random graph model that allows us to deal with the quenched disorder. Therefore, the connectivity becomes a controllable parameter in our theory, allowing us to answer what the differences are between this description and the mean-field theory i.e., the fully connected theory. We have considered the random network random field Ising model where the spin exchange interaction as well as the RF are random variables following a Gaussian distribution. The results were found within the replica symmetric (RS) approximation, whose stability is obtained using the two-replica method. This also puts our work in the context of a broader discussion, which is the RS stability as a function of the connectivity. In particular, our results show that for small connectivity there is a region at zero temperature where the RS solution remains stable above a given value of the magnetic field no matter the strength of RF. Consequently, our results show important differences with the crossover between the RF and SG regimes predicted by the fully connected theory.

Solving the spherical p -spin model with the cavity method: equivalence with the replica results

The spherical p-spin is a fundamental model for glassy physics, thanks to its analytical solution achievable via the replica method. Unfortunately, the replica method has some drawbacks: it is very hard to apply to diluted models and the assumptions beyond it are not immediately clear. Both drawbacks can be overcome by the use of the cavity method; however, this needs to be applied with care to spherical models. Here, we show how to write the cavity equations for spherical p-spin models, both in the replica symmetric (RS) ansatz (corresponding to belief propagation) and in the one-step replica-symmetry-breaking (1RSB) ansatz (corresponding to survey propagation). The cavity equations can be solved by a Gaussian RS and multivariate Gaussian 1RSB ansatz for the distribution of the cavity fields. We compute the free energy in both ansatzes and check that the results are identical to the replica computation, predicting a phase transition to a 1RSB phase at low temperatures. The advantages of solving the model with the cavity method are many. The physical meaning of the ansatz for the cavity marginals is very clear. The cavity method works directly with the distribution of local quantities, which allows us to generalize the method to diluted graphs. What we are presenting here is the first step towards the solution of the diluted version of the spherical p-spin model, which is a fundamental model in the theory of random lasers and interesting per se as an easier-to-simulate version of the classical fully connected p-spin model.

Compressed Sensing Performance Analysis Via Replica Method Using Bayesian Framework

Compressive sensing (CS) is a new methodology to capture signals at lower rate than the Nyquist sampling rate when the signals are sparse or sparse in some domain. The performance of CS estimators is analyzed in this paper using tools from statistical mechanics, especially called replica method. This method has been used to analyze communication systems like Code Division Multiple Access (CDMA) and multiple input multi- ple output (MIMO) systems with large size. Replica analysis, now days rigorously proved, is an efficient tool to analyze large systems in general. Specifically, we analyze the performance of some of the estimators used in CS like LASSO (the Least Absolute Shrinkage and Selection Operator) estimator and Zero-Norm regularizing estimator as a special case of maximum a posteriori (MAP) estimator by using Bayesian framework to connect the CS estimators and replica method. We use both replica symmetric (RS) ansatz and one-step replica symmetry breaking (1RSB) ansatz, clamming the latter is efficient when the problem is not convex. This work is more analytical in its form. It is deferred for next step to focus on the numerical results.

Approximate message passing for nonconvex sparse regularization with stability and asymptotic analysis

We analyse a linear regression problem with nonconvex regularization called smoothly clipped absolute deviation (SCAD) under an overcomplete Gaussian basis for Gaussian random data. We propose an approximate message passing (AMP) algorithm considering nonconvex regularization, namely SCAD-AMP, and analytically show that the stability condition corresponds to the de Almeida–Thouless condition in spin glass literature. Through asymptotic analysis, we show the correspondence between the density evolution of SCAD-AMP and the replica symmetric (RS) solution. Numerical experiments confirm that for a sufficiently large system size, SCAD-AMP achieves the optimal performance predicted by the replica method. Through replica analysis, a phase transition between replica symmetric and replica symmetry breaking (RSB) region is found in the parameter space of SCAD. The appearance of the RS region for a nonconvex penalty is a significant advantage that indicates the region of smooth landscape of the optimization problem. Furthermore, we analytically show that the statistical representation performance of the SCAD penalty is better than that of -based methods, and the minimum representation error under RS assumption is obtained at the edge of the RS/RSB phase. The correspondence between the convergence of the existing coordinate descent algorithm and RS/RSB transition is also indicated.

Evaluation of generalized degrees of freedom for sparse estimation by replica method

We develop a method to evaluate the generalized degrees of freedom (GDF) for linear regression with sparse regularization. The GDF is a key factor in model selection, and thus its evaluation is useful in many modelling applications. An analytical expression for the GDF is derived using the replica method in the large-system-size limit with random Gaussian predictors. The resulting formula has a universal form that is independent of the type of regularization, providing us with a simple interpretation. Within the framework of replica symmetric (RS) analysis, GDF has a physical meaning as the effective fraction of non-zero components. The validity of our method in the RS phase is supported by the consistency of our results with previous mathematical results. The analytical results in the RS phase are calculated numerically using the belief propagation algorithm.

A spin glass approach to the directed feedback vertex set problem

A directed graph (digraph) is formed by vertices and arcs (directed edges) from one vertex to another. A feedback vertex set (FVS) is a set of vertices that contains at least one vertex of every directed cycle in this digraph. The directed feedback vertex set problem aims at constructing a FVS of minimum cardinality. This is a fundamental cycle-constrained hard combinatorial optimization problem with wide practical applications. In this paper we construct a spin glass model for the directed FVS problem by converting the global cycle constraints into local arc constraints, and study this model through the replica-symmetric (RS) mean field theory of statistical physics. We then implement a belief propagation-guided decimation (BPD) algorithm for single digraph instances. The BPD algorithm slightly outperforms the simulated annealing algorithm on large random graph instances. The RS mean field results and algorithmic results can be further improved by working on a more restrictive (and more difficult) spin glass model.

Supervised Learning of Two-Layer Perceptron under the Existence of External Noise — Learning Curve of Boolean Functions of Two Variables in Tree-Like Architecture —

We investigate the supervised batch learning of Boolean functions expressed by a two-layer perceptron with a tree-like structure. We adopt continuous weights (spherical model) and the Gibbs algorithm. We study the Parity and And machines and two types of noise, input and output noise, together with the noiseless case. We assume that only the teacher suffers from noise. By using the replica method, we derive the saddle point equations for order parameters under the replica symmetric (RS) ansatz. We study the critical value αC of the loading rate α above which the learning phase exists for cases with and without noise. We find that αC is nonzero for the Parity machine, while it is zero for the And machine. We derive the exponents \(\bar{\beta}\) of order parameters expressed as \((\alpha - \alpha _{\text{C}})^{\bar{\beta}}\) when α is near to αC. Furthermore, in the Parity machine, when noise exists, we find a spin glass solution, in which the overlap between the teacher and student vectors is zero but that...

Some properties of the phase diagram for mixed p-spin glasses

In this paper we study the Parisi variational problem for mixed $p$-spin glasses with Ising spins. Our starting point is a characterization of Parisi measures whose origin lies in the first order optimality conditions for the Parisi functional, which is known to be strictly convex. Using this characterization, we study the phase diagram in the temperature-external field plane. We begin by deriving self-consistency conditions for Parisi measures that generalize those of de Almeida and Thouless to all levels of Replica Symmetry Breaking (RSB) and all models. As a consequence, we conjecture that for all models the Replica Symmetric (RS) phase is the region determined by the natural analogue of the de Almeida-Thouless condition. We show that for all models, the complement of this region is in the RSB phase. Furthermore, we show that the conjectured phase boundary is exactly the phase boundary in the plane less a bounded set. In the case of the Sherrington-Kirkpatrick model, we extend this last result to show that this bounded set does not contain the critical point at zero external field.

Effect of modularity on the Glauber dynamics of the dilute spin glass model

We study the Glauber dynamics of the dilute, infinite-ranged spin glass model, the so-called dilute Sherrington-Kirkpatrick (dSK) model. The dSK model has sparse couplings and can be classified by the modularity (M) of the coupling matrix. We investigate the effect of the modularity on the relaxation dynamics starting from a random initial state. By using the Glauber dynamics and the replica method, we derive the relaxation dynamics equations for the magnetization (m) and the energy per spin (r), in addition to the equation for the spin glass order parameter (q αβ ). In the replica symmetric (RS) analysis, we find that there are two solutions for the RS spin glass order parameter (q): q = 0which is stable for r 1/2 in the non-modular system and q = 0 which is stable for r 1/\(\sqrt 8 \) in the completely modular system. By substituting the proper q values into the equations for r, we find that the relaxation dynamics of r depends on the modularity, M. These results suggest that, in the context of evolutionary theory, the modularity may emerge spontaneously in the point-mutation-only framework (Glauber dynamics) under a changing environment.

Learning from Stochastic Rules by Spherical Perceptrons under Finite Temperature –Optimal Temperature and Asymptotic Learning Curve–

In the problem of learning under external disturbance, there is a possibility that the existence of some tolerance or flexibility in the system weakens the effect of noise and helps the system to perform more efficiently. In a previous letter, we gave one example of such phenomena in learning from stochastic rules by spherical perceptrons adopting the Gibbs algorithm using statistical mechanical methods. By the replica method, we showed that, in the output noise model, there exists an optimal temperature at which the generalization error takes its minimum for the stable replica symmetric (RS) solution. On the other hand, for other types of noise including input noise, it was shown that no such temperature exists up to the one-step replica symmetry breaking (1RSB) solution. That is, it was shown that for the asymptotic region of a large number of training sets, the RS solution becomes unstable, and the asymptotic behavior is determined by the 1RSB solution, The asymptotic expressions for learning curves we...

Potts spin glasses with three, four and five states near T = Tc: expanding around the replica symmetric solution

Expansion for the 1RSB free energy functionals of the mean-field (MF) Potts spin glass models up to the fourth order in δqαβ around the replica symmetric (RS) solution is investigated. We consider models with three, four and five states and use a special quadrupole-like representation for them. The corresponding series for the 1RSB order parameters giving explicit temperature dependence are obtained in the vicinity of the point T = Tc, where the RS solution becomes unstable. The crossover from continuous to jumpwise behavior with increasing number of states is derived analytically. A comparison is made between the free energy expansion for the Potts spin glass and that for other models.

Optimal location of sources in transportation networks

We consider the problem of optimizing the locations of source nodes in transportationnetworks. A reduction of the fraction of surplus nodes induces a glassy transition. Incontrast to most constraint satisfaction problems involving discrete variables, ourproblem involves continuous variables which lead to cavity fields in the form offunctions. The one-step replica symmetry breaking (1RSB) solution involves solving astable distribution of functionals, which is in general infeasible. In this paper, weobtain small closed sets of functional cavity fields and demonstrate how functionalrecursions are converted to simple recursions of probabilities which make the 1RSBsolution feasible. The physical results in the replica symmetric (RS) and the 1RSBframeworks are thus derived and the stabilities of the RS and 1RSB solutions areexamined.

The Sherrington–Kirkpatrick model near T = Tc: expanding around the replica symmetric solution

An expansion for the free energy functional of the Sherrington–Kirkpatrick (SK) model, around the replica symmetric (RS) SK solution Q(RS)ab = δab + q(1 − δab) is investigated. In particular, when the expansion is truncated to the fourth order in Qab − Q(RS)ab the full replica symmetry broken (FRSB) solution is explicitly found but it turns out to exist only in the range of temperature 0.549… ⩽ T ⩽ Tc = 1, not including T = 0. On the other hand, an expansion around the paramagnetic solution Q(PM)ab = δab, up to the fourth order, yields a FRSB solution that exists in a limited temperature range 0.915… ⩽ T ⩽ Tc = 1.

Stability analysis on the finite-temperature replica-symmetric and first-step replica-symmetry-broken cavity solutions of the random vertex cover problem

The vertex cover problem is a prototypical hard combinatorial optimization problem. It was studied in recent years by physicists using the cavity method of statistical mechanics. In this paper, the stability of the finite-temperature replica-symmetric (RS) and the first-step replica-symmetry-broken (1RSB) cavity solutions of the vertex cover problem on random regular graphs of finite vertex degree K are analyzed by population dynamics simulations. We found that (1) the lowest temperature for the RS solution to be stable, T(RS)(K), is not a monotonic function of K; (2) at relatively large connectivity K and temperature T slightly below the dynamic transition temperature T(d)(K), the 1RSB solutions with small but non-negative complexity values are stable, and (3) the dynamical transition temperature T(d) and Kauzmann temperature T(K) is equal to each other. Similar results are obtained on random Poissonian graphs.

Zero temperature solutions of the Edwards-Anderson model in random Husimi lattices

We solve the Edwards-Anderson model (EA) in different Husimi lattices using the cavity method at replica symmetric (RS) and 1-step of replica symmetry breaking (1RSB) levels. We show that, at T = 0, the structure of the solution space depends on the parity of the loop sizes. Husimi lattices with odd loop sizes may have a trivial paramagnetic solution thermodynamically relevant for highly frustrated systems while, in Husimi lattices with even loop sizes, this solution is absent. The range of stability under 1RSB perturbations of this and other RS solutions is computed analytically (when possible) or numerically. We also study the transition from 1RSB solutions to paramagnetic and ferromagnetic RS solutions. Finally we compare the solutions of the EA model in Husimi lattices with that on the (short loops free) Bethe lattices, showing that already for loop sizes of order 8 both models behave similarly.