Limit Of Large Networks Research Articles

Les Houches lectures on deep learning at large and infinite width*

Abstract These lectures, presented at the 2022 Les Houches Summer School on Statistical Physics and Machine Learning, focus on the infinite-width limit and large-width regime of deep neural networks. Topics covered include the various statistical and dynamical properties of these networks. In particular, the lecturers discuss properties of random deep neural networks, connections between trained deep neural networks, linear models, kernels and Gaussian processes that arise in the infinite-width limit, and perturbative and non-perturbative treatments of large but finite-width networks, at initialization and after training.

Complex networks of interacting stochastic tipping elements: Cooperativity of phase separation in the large-system limit.

Tipping elements in the Earth system have received increased scientific attention over recent years due to their nonlinear behavior and the risks of abrupt state changes. While being stable over a large range of parameters, a tipping element undergoes a drastic shift in its state upon an additional small parameter change when close to its tipping point. Recently, the focus of research broadened towards emergent behavior in networks of tipping elements, like global tipping cascades triggered by local perturbations. Here, we analyze the response to the perturbation of a single node in a system that initially resides in an unstable equilibrium. The evolution is described in terms of coupled nonlinear equations for the cumulants of the distribution of the elements. We show that drift terms acting on individual elements and offsets in the coupling strength are subdominant in the limit of large networks, and we derive an analytical prediction for the evolution of the expectation (i.e., the first cumulant). It behaves like a single aggregated tipping element characterized by a dimensionless parameter that accounts for the network size, its overall connectivity, and the average coupling strength. The resulting predictions are in excellent agreement with numerical data for Erdös-Rényi, Barabási-Albert, and Watts-Strogatz networks of different size and with different coupling parameters.

Degree-ordered-percolation on uncorrelated networks

We analyze the properties of degree-ordered percolation (DOP), a model in which the nodes of a network are occupied in degree-descending order. This rule is the opposite of the much studied degree-ascending protocol, used to investigate resilience of networks under intentional attack, and has received limited attention so far. The interest in DOP is also motivated by its connection with the susceptible-infected-susceptible (SIS) model for epidemic spreading, since a variation of DOP is related to the vanishing of the SIS transition for random power-law degree-distributed networks P(k) ∼ k −γ . By using the generating function formalism, we investigate the behavior of the DOP model on networks with generic value of γ and we validate the analytical results by means of numerical simulations. We find that the percolation threshold vanishes in the limit of large networks for γ ⩽ 3, while it is finite for γ > 3, although its value for γ between 3 and 4 is exceedingly small and preasymptotic effects are huge. We also derive the critical properties of the DOP transition, in particular how the exponents depend on the heterogeneity of the network, determining that DOP does not belong to the universality class of random percolation for γ ⩽ 3.

On Fundamental Limitations of Dynamic Feedback Control in Regular Large-Scale Networks

In this paper, we study fundamental performance limitations of distributed feedback control in large-scale networked dynamical systems. Specifically, we address the question of whether dynamic feedback controllers perform better than static (memoryless) ones when subject to locality constraints. We consider distributed linear consensus and vehicular formation control problems modeled over toric lattice networks. For the resulting spatially invariant systems, we study the large-scale asymptotics (in network size) of global performance metrics that quantify the level of network coherence. With static feedback from relative state measurements, such metrics are known to scale unfavorably in lattices of low spatial dimensions, preventing, for example, a one-dimensional string of vehicles to move like a rigid object. We show that the same limitations in general apply also to dynamic feedback control that is locally of first order. This means that the addition of one local state to the controller gives a similar asymptotic performance to the memoryless case. This holds unless the controller can access noiseless measurements of its local state with respect to an absolute reference frame, in which case the addition of controller memory may fundamentally improve performance. In simulations of platoons with 20–200 vehicles, we show that the performance limitations we derive manifest as unwanted accordionlike motions. Similar behaviors are to be expected in any network that is embeddable in a low-dimensional toric lattice, and the same fundamental limitations would apply. To derive our results, we present a general technical framework for the analysis of stability and performance of spatially invariant systems in the limit of large networks.

A Re-entrant Phase Transition in the Survival of Secondary Infections on Networks

We study the dynamics of secondary infections on networks, in which only the individuals currently carrying a certain primary infection are susceptible to the secondary infection. In the limit of large sparse networks, the model is mapped to a branching process spreading in a random time-sensitive environment, determined by the dynamics of the underlying primary infection. When both epidemics follow the Susceptible-Infective-Recovered model, we show that in order to survive, it is necessary for the secondary infection to evolve on a timescale that is closely matched to that of the primary infection on which it depends.

Are network properties consistent indicators of synchronization?

We investigate the collective dynamics of bi-stable elements connected in different network topologies, ranging from rings and small-world networks, to random and deterministic scale-free networks. We focus on the correlation between network properties and global stability measures of the synchronized state, in particular the average critical coupling strength yielding transition to synchronization. Further, we estimate the robustness of the synchronized state by finding the minimal fraction of nodes fc that need to be perturbed in order to lose synchronization. Our central result from these synchronization features is the following: while networks properties can provide indicators of synchronization within a network class, they fail to provide consistent indicators across network classes. For instance, the heterogeneity of degree does not consistently impact synchronization, as is evident through the stark difference in the synchronizability of rings vis-à-vis small-world and star networks, all of which have same average degree and deviation around the mean degree in the limit of large networks. Further we demonstrate that clustering coefficient is also not a consistent feature in determining synchronization. This is clear through the similarity of synchronization properties in rings with significantly different clustering coefficients, and the striking difference in synchronization of a star network and a ring having the same clustering coefficient. Even the characteristic path length, which is of paramount importance in determining synchronization, does not provide a one-to-one correspondence with synchronization properties across classes. Namely, synchronization is significantly favoured in networks with low path lengths within a network class. However, the same characteristic path length in different types of networks yields very different and fc.

Coexistence in Preferential Attachment Networks

We introduce a new model of competition on growing networks. This extends the preferential attachment model, with the key property that node choices evolve simultaneously with the network. When a new node joins the network, it chooses neighbours by preferential attachment, and selects its type based on the number of initial neighbours of each type. The model is analysed in detail, and in particular, we determine the possible proportions of the various types in the limit of large networks. An important qualitative feature we find is that, in contrast to many current theoretical models, often several competitors will coexist. This matches empirical observations in many real-world networks.

Speed, dissipation, and error in kinetic proofreading

Proofreading mechanisms increase specificity in biochemical reactions by allowing for the dissociation of intermediate complexes. These mechanisms disrupt and reset the reaction to undo errors at the cost of increased time of reaction and free energy expenditure. Here, we draw an analogy between proofreading and microtubule growth which share some of the features described above. Our analogy relates the statistics of growth and shrinkage of microtubules in physical space to the cycling of intermediate complexes in the space of chemical states in proofreading mechanisms. Using this analogy, we find a new kinetic regime of proofreading in which an exponential speed-up of the process can be achieved at the cost of a somewhat larger error rate. This regime is analogous to the transition region between two known growth regimes of microtubules (bounded and unbounded) and is sharply defined in the limit of large proofreading networks. We find that this advantageous regime of speed-error tradeoff might be present in proofreading schemes studied earlier in the charging of tRNA by tRNA synthetases, in RecA filament assembly on ssDNA, and in protein synthesis by ribosomes.

Optimal learning rules for familiarity detection

It has been suggested that the mammalian memory system has both familiarity and recollection components. Recently, a high-capacity network to store familiarity has been proposed. Here we derive analytically the optimal learning rule for such a familiarity memory using a signal- to-noise ratio analysis. We find that in the limit of large networks the covariance rule, known to be the optimal local, linear learning rule for pattern association, is also the optimal learning rule for familiarity discrimination. In the limit of large networks, the capacity is independent of the sparseness of the patterns and the corresponding information capacity is 0.057 bits per synapse, which is somewhat less than typically found for associative networks.

Connection between scale-free networks and nonextensive statistical mechanics

The degree distribution of the so-called scale-free networks exhibits, quite often, the form p(k) ∝1/(k0+k)γ (with γ>0 and k0>0), in the limit of large networks. It happens that this form precisely coincides with the q-exponential p(k) ∝exp q(-k/κ) (q ≥1 and κ>0), with γ=1/(q-1) and k0=κ/(q-1). It optimises the nonadditive entropy \(S_q= \frac{k_B}{q-1}\left\{ 1-\sum_{k=1}^\infty [p(k)]^q \right\}\) with mathematically the same constraints that yield the stationary (or quasi-stationary) distribution in nonextensive statistical mechanics. In other words, the most ubiquitous form of the degree distribution of scale-free networks is a realisation of the hypothesis involved within the q-generalisation of Boltzmann-Gibbs statistical mechanics. In addition to this, we show that growth is not a necessary condition for having scale-free networks, in contrast with a widely spread belief.

Self-avoiding walks on scale-free networks

Several kinds of walks on complex networks are currently used to analyze search and navigation in different systems. Many analytical and computational results are known for random walks on such networks. Self-avoiding walks (SAW's) are expected to be more suitable than unrestricted random walks to explore various kinds of real-life networks. Here we study long-range properties of random SAW's on scale-free networks, characterized by a degree distribution P(k) approximately k(-gamma). In the limit of large networks (system size N-->infinity), the average number sn of SAW's starting from a generic site increases as mu(n) , with mu = k2/k-1 . For finite N, sn is reduced due to the presence of loops in the network, which causes the emergence of attrition of the paths. For kinetic growth walks, the average maximum length L increases as a power of the system size: L approximately Nalpha, with an exponent alpha increasing as the parameter gamma is raised. We discuss the dependence of alpha on the minimum allowed degree in the network. A similar power-law dependence is found for the mean self-intersection length of nonreversal random walks. Simulation results support our approximate analytical calculations.

Conductance in random inductance-capacitance networks.

We consider the admittance Y of large square networks in which each link is randomly, and with equal probability, assigned a component of inductance L or capacitance C. When a potential sin(\ensuremath{\omega}t) is applied over the network, Y(\ensuremath{\omega}) is purely imaginary and alternates, as a function of \ensuremath{\omega}, between inductive and capacitive behavior, with resonances at frequencies ${\mathrm{\ensuremath{\omega}}}_{\mathit{i}}$, which we find numerically. The corresponding frequency-density function is close to that of the logistic distribution in mathematical statistics. In the limit of large networks, the fraction 0.1964\ifmmode\pm\else\textpm\fi{}0.0007 of all ${\mathrm{\ensuremath{\omega}}}_{\mathit{i}}$ are 0 or \ensuremath{\infty}. That fraction is related to the number of L and C clusters in the network. We also discuss the amplitude of the potential at the nodes of the lattice, i.e., the eigenstates corresponding to the eigenvalues ${\mathrm{\ensuremath{\omega}}}_{\mathit{i}}$.