Function Of Network Size Research Articles

Graceful Degradation of Recurrent Neural Networks as a Function of Network Size, Memory Length, and Connectivity Damage

Recurrent Neural Networks (RNNs) are frequently used to model aspects of brain function and structure. In this work, we trained small fully-connected RNNs to perform temporal and flow control tasks with time-varying stimuli. Our results show that different RNNs can solve the same task by converging to different underlying dynamics and that the performance gracefully degrades either as network size is decreased, interval duration is increased, or connectivity is damaged. Our results are useful to quantify different aspects of the models, which are normally used as black boxes and need to be understood in advance to modeling the biological response of cerebral cortex areas.

Gated Linear Networks

This paper presents a new family of backpropagation-free neural architectures, Gated Linear Networks (GLNs). What distinguishes GLNs from contemporary neural networks is the distributed and local nature of their credit assignment mechanism; each neuron directly predicts the target, forgoing the ability to learn feature representations in favor of rapid online learning. Individual neurons are able to model nonlinear functions via the use of data-dependent gating in conjunction with online convex optimization. We show that this architecture gives rise to universal learning capabilities in the limit, with effective model capacity increasing as a function of network size in a manner comparable with deep ReLU networks. Furthermore, we demonstrate that the GLN learning mechanism possesses extraordinary resilience to catastrophic forgetting, performing almost on par to an MLP with dropout and Elastic Weight Consolidation on standard benchmarks.

The Pseudomonas aeruginosa whole genome sequence: A 20th anniversary celebration.

Toward the end of August 2000, the 6.3 Mbp whole genome sequence of Pseudomonas aeruginosa strain PAO1 was published. With 5570 open reading frames (ORFs), PAO1 had the largest microbial genome sequenced up to that point in time-including a large proportion of metabolic, transport and antimicrobial resistance genes supporting its ability to colonize diverse environments. A remarkable 9% of its ORFs were predicted to encode proteins with regulatory functions, providing new insight into bacterial network complexity as a function of network size. In this celebratory article, we fast forward 20 years, and examine how access to this resource has transformed our understanding of P. aeruginosa. What follows is more than a simple review or commentary; we have specifically asked some of the leaders in the field to provide personal reflections on how the PAO1 genome sequence, along with the Pseudomonas Community Annotation Project (PseudoCAP) and Pseudomonas Genome Database (pseudomonas.com), have contributed to the many exciting discoveries in this field. In addition to bringing us all up to date with the latest developments, we also ask our contributors to speculate on how the next 20 years of Pseudomonas research might pan out.

Self-supervised learning with physics-aware neural networks – I. Galaxy model fitting

ABSTRACT Estimating the parameters of a model describing a set of observations using a neural network is, in general, solved in a supervised way. In cases when we do not have access to the model’s true parameters, this approach can not be applied. Standard unsupervised learning techniques, on the other hand, do not produce meaningful or semantic representations that can be associated with the model’s parameters. Here we introduce a novel self-supervised hybrid network architecture that combines traditional neural network elements with analytic or numerical models, which represent a physical process to be learned by the system. Self-supervised learning is achieved by generating an internal representation equivalent to the parameters of the physical model. This semantic representation is used to evaluate the model and compare it to the input data during training. The semantic autoencoder architecture described here shares the robustness of neural networks while including an explicit model of the data, learns in an unsupervised way, and estimates, by construction, parameters with direct physical interpretation. As an illustrative application, we perform unsupervised learning for 2D model fitting of exponential light profiles and evaluate the performance of the network as a function of network size and noise.

Edge crossings in random linear arrangements

In spatial networks vertices are arranged in some space and edges may cross. When arranging vertices in a 1D lattice edges may cross when drawn above the vertex sequence as it happens in linguistic and biological networks. Here we investigate the general problem of the distribution of edge crossings in random arrangements of the vertices. We generalize the existing formula for the expectation of this number in random linear arrangements of trees to any network and derive an expression for the variance of the number of crossings in an arbitrary layout relying on a novel characterization of the algebraic structure of that variance in an arbitrary space. We provide compact formulae for the expectation and the variance in complete graphs, complete bipartite graphs, cycle graphs, one-regular graphs and various kinds of trees (star trees, quasi-star trees and linear trees). In these networks, the scaling of expectation and variance as a function of network size is asymptotically power-law-like in random linear arrangements. Our work paves the way for further research and applications in one-dimension or investigating the distribution of the number of crossings in lattices of higher dimension or other embeddings.

Efficient content retrieval in fog zone using Nano‐Caches

SummaryIt is always desired to improve the response time from cloud servers, which deliver contents without buffering. As the penetration of mobile/fog devices is increasing, the limits of cellular ranges come under question. This question arises in spite of the fact that the current Internet Service Providers and data operators are adding cellular towers frequently to reduce delay and enhance performance. This performance can be improved by increasing Nano‐Cache(s) at the edges of the network for forwarding interrelated contents to remote corner of the earth. In this research work, Nano‐Caches are integrated for delivering contents efficiently, using search‐based optimization techniques, which are energy and response aware in nature. An algorithm, namely, Modified Teaching Learning‐Based Optimization(MTLBO), is devised and implemented in fog zone to find efficient route for forwarding contents using Nano‐Caches and subsequently to improve content retrieval time. Mathematical distribution model of traffic is used for simulation process. MTLBO is compared with existing algorithms, namely, Teaching Learning‐Based Optimization (TLBO) Algorithm and Simulated Annealing (SA) Algorithm. The design of experiments (DOE) was carried out to observe number of iterations, learning rate, and by changing the network size. Java library was used for observing values of memory and execution time. The results show that Modified Teaching Learning‐Based Optimization (MTLBO) approach is better than Teaching Learning‐Based Optimization (TLBO) approach as it has less overheads in terms of memory (considering number of fog caches) and network size for delivering contents at remote areas. In comparison to the Simulated Annealing (SA) algorithm, MTLBO performs better in terms of execution time, overhead in terms of memory, and scalability as function of network size.

RANDOM WALKS WITH A TRAP IN SCALE-FREE FRACTAL HIERARCHICAL LATTICES

In this paper, we study two kinds of random walks with a trap in a class of scale-free fractal hierarchical lattices. One is standard random walks belonging to unbiased random walks, while the other one named mixed random walks is biased. The structural properties of these hierarchical lattices are controlled by a parameter [Formula: see text]. We derive exact solutions of the average trapping time (ATT) for the two trapping issue, respectively. The results show that in large networks, both of the ATT grow asymptotically as a power-law function of network size with the exponent related to the parameter [Formula: see text]. It indicates that network structure has a substantial effect on the efficiency of trapping processes performed in scale-free networks. Comparing the results obtained for the two different random walks, we find that changes of the walking rule have no effect on the leading exponent of the ATT, but could modify the coefficient of the formula for the ATT. The findings are helpful for better understanding the influence factor of random walks in complex systems.

Staleness Bounds and Efficient Protocols for Dissemination of Global Channel State Information

This paper considers the problem of achieving global channel knowledge throughout a fully connected packetized wireless network with time-varying channels. While the value of channel state information at the transmitter (CSIT) is now well known, there are many scenarios in which it is helpful to have additional channel knowledge beyond conventional CSIT, e.g., cooperative communication systems. The overhead required for global CSI knowledge can be significant, particularly in time-varying channels where the quality of channel estimates is dominated by the “staleness” of the CSI. Nevertheless, the fundamental limits and feasibility of tracking global CSI throughout a network have not been sufficiently studied. This paper presents a framework for analyzing the staleness of protocols that estimate and disseminate CSI to all nodes in a fully connected network. Fundamental bounds on achievable staleness are derived, and efficient dissemination protocols are developed, which achieve these limits. The results provide engineering guidelines on the feasibility of tracking global CSI as a function of network size, the size and composition of the packets, packet error rate, and channel coherence time.

Leaking privacy and shadow profiles in online social networks.

Social interaction and data integration in the digital society can affect the control that individuals have on their privacy. Social networking sites can access data from other services, including user contact lists where nonusers are listed too. Although most research on online privacy has focused on inference of personal information of users, this data integration poses the question of whether it is possible to predict personal information of nonusers. This article tests the shadow profile hypothesis, which postulates that the data given by the users of an online service predict personal information of nonusers. Using data from a disappeared social networking site, we perform a historical audit to evaluate whether personal data of nonusers could have been predicted with the personal data and contact lists shared by the users of the site. We analyze personal information of sexual orientation and relationship status, which follow regular mixing patterns in the social network. Going back in time over the growth of the network, we measure predictor performance as a function of network size and tendency of users to disclose their contact lists. This article presents robust evidence supporting the shadow profile hypothesis and reveals a multiplicative effect of network size and disclosure tendencies that accelerates the performance of predictors. These results call for new privacy paradigms that take into account the fact that individual privacy decisions do not happen in isolation and are mediated by the decisions of others.

Cut-Set Bounds on Network Information Flow

Explicit characterization of the capacity region of communication networks is a long standing problem. While it is known that network coding can outperform routing and replication, the set of feasible rates is not known in general. Characterizing the network coding capacity region requires determination of the set of all entropic vectors. Furthermore, computing the explicitly known linear programming bound is infeasible in practice due to an exponential growth in complexity as a function of network size. This paper focuses on the fundamental problems of characterization and computation of outer bounds for networks with correlated sources. Starting from the known local functional dependencies induced by the communications network, we introduce the notion of irreducible sets, which characterize implied functional dependencies. We provide recursions for computation of all maximal irreducible sets. These sets act as information-theoretic bottlenecks, and provide an easily computable outer bound. We extend the notion of irreducible sets (and resulting outer bound) for networks with independent sources. We compare our bounds with existing bounds in the literature. We find that our new bounds are the best among the known graph theoretic bounds for networks with correlated sources and for networks with independent sources.

Stochastic mean-field formulation of the dynamics of diluted neural networks

We consider pulse-coupled leaky integrate-and-fire neural networks with randomly distributed synaptic couplings. This random dilution induces fluctuations in the evolution of the macroscopic variables and deterministic chaos at the microscopic level [1]. Our main aim is to mimic the effect of the dilution as a noise source acting on the dynamics of a globally coupled nonchaotic system. We show that the evolution of a deterministic diluted neural network of any size can be well approximated by a much smaller fully coupled network, where each neuron is driven by a mean synaptic current plus additive noise. These terms represent the average and the fluctuations of the synaptic currents acting on the single neurons in the diluted system. The main microscopic and macroscopic dynamical features can be reproduced within this stochastic approximation. In order to illustrate the quality of this reconstruction, we compare the probability distribution function of the inter-spike intervals and the macroscopic attractor of the deterministic diluted system with those obtained by employing the mean-field stochastic model (see Figure ​Figure1A1A and ​and1B,1B, respectively). Furthermore, the microscopic stability of the diluted network can be also reproduced, as demonstrated from the almost coincidence of the measured Lyapunov exponents in the deterministic and stochastic cases for an ample range of system sizes (see Figure ​Figure1C.).1C.). Our results strongly suggest that the fluctuations in the synaptic currents are responsible for the emergence of chaos in this class of pulse-coupled systems [2]. Figure 1 A. Inter-spike interval distribution functions and B. attractor dynamics for a representative case of dilution with system size ND = 5000. C. Maximal Lyapunov exponent as a function of network size. In all panels, black circled symbols indicate the diluted ...

Simulation study of large production network robustness in uncertain environment

Robustness is an important success factor for production networks in which the operation of enterprises is subjected to an uncertain environment. In this paper, the robustness of networks is studied as a function of network size. The study is performed through a simulation experiment in which the uncertain environment is modelled by introducing perturbations in demand. The decision-making model mimics the behaviour of socially connected human subjects. The results show how robustness and production rate are affected by system size and social network structure, and how this is relevant for the design and operation of future manufacturing systems.

To cut or not to cut? Assessing the modular structure of brain networks

A wealth of methods has been developed to identify natural divisions of brain networks into groups or modules, with one of the most prominent being modularity. Compared with the popularity of methods to detect community structure, only a few methods exist to statistically control for spurious modules, relying almost exclusively on resampling techniques. It is well known that even random networks can exhibit high modularity because of incidental concentration of edges, even though they have no underlying organizational structure. Consequently, interpretation of community structure is confounded by the lack of principled and computationally tractable approaches to statistically control for spurious modules. In this paper we show that the modularity of random networks follows a transformed version of the Tracy–Widom distribution, providing for the first time a link between module detection and random matrix theory. We compute parametric formulas for the distribution of modularity for random networks as a function of network size and edge variance, and show that we can efficiently control for false positives in brain and other real-world networks.

Intrinsic Noise Induces Critical Behavior in Leaky Markovian Networks Leading to Avalanching

The role intrinsic statistical fluctuations play in creating avalanches – patterns of complex bursting activity with scale-free properties – is examined in leaky Markovian networks. Using this broad class of models, we develop a probabilistic approach that employs a potential energy landscape perspective coupled with a macroscopic description based on statistical thermodynamics. We identify six important thermodynamic quantities essential for characterizing system behavior as a function of network size: the internal potential energy, entropy, free potential energy, internal pressure, pressure, and bulk modulus. In agreement with classical phase transitions, these quantities evolve smoothly as a function of the network size until a critical value is reached. At that value, a discontinuity in pressure is observed that leads to a spike in the bulk modulus demarcating loss of thermodynamic robustness. We attribute this novel result to a reallocation of the ground states (global minima) of the system's stationary potential energy landscape caused by a noise-induced deformation of its topographic surface. Further analysis demonstrates that appreciable levels of intrinsic noise can cause avalanching, a complex mode of operation that dominates system dynamics at near-critical or subcritical network sizes. Illustrative examples are provided using an epidemiological model of bacterial infection, where avalanching has not been characterized before, and a previously studied model of computational neuroscience, where avalanching was erroneously attributed to specific neural architectures. The general methods developed here can be used to study the emergence of avalanching (and other complex phenomena) in many biological, physical and man-made interaction networks.

Stochastic solution of population balance equations for reactor networks

This work presents a sequential modular approach to solve a generic network of reactors with a population balance model using a stochastic numerical method. Full-coupling to the gas-phase is achieved through operator-splitting. The convergence of the stochastic particle algorithm in test networks is evaluated as a function of network size, recycle fraction and numerical parameters. These test cases are used to identify methods through which systematic and statistical error may be reduced, including by use of stochastic weighted algorithms. The optimal algorithm was subsequently used to solve a one-dimensional example of silicon nanoparticle synthesis using a multivariate particle model. This example demonstrated the power of stochastic methods in resolving particle structure by investigating the transient and spatial evolution of primary polydispersity, degree of sintering and TEM-style images.

Binary Willshaw learning yields high synaptic capacity for long-term familiarity memory

In this study, we investigate from a computational perspective the efficiency of the Willshaw synaptic update rule in the context of familiarity discrimination, a binary-answer, memory-related task that has been linked through psychophysical experiments with modified neural activity patterns in the prefrontal and perirhinal cortex regions. Our motivation for recovering this well-known learning prescription is two-fold: first, the switch-like nature of the induced synaptic bonds, as there is evidence that biological synaptic transitions might occur in a discrete stepwise fashion. Second, the possibility that in the mammalian brain, unused, silent synapses might be pruned in the long-term. Besides the usual pattern and network capacities, we calculate the synaptic capacity of the model, a recently proposed measure where only the functional subset of synapses is taken into account. We find that in terms of network capacity, Willshaw learning is strongly affected by the pattern coding rates, which have to be kept fixed and very low at any time to achieve a non-zero capacity in the large network limit. The information carried per functional synapse, however, diverges and is comparable to that of the pattern association case, even for more realistic moderately low activity levels that are a function of network size.

Meeting the Memory Challenges of Brain-Scale Network Simulation

The development of high-performance simulation software is crucial for studying the brain connectome. Using connectome data to generate neurocomputational models requires software capable of coping with models on a variety of scales: from the microscale, investigating plasticity, and dynamics of circuits in local networks, to the macroscale, investigating the interactions between distinct brain regions. Prior to any serious dynamical investigation, the first task of network simulations is to check the consistency of data integrated in the connectome and constrain ranges for yet unknown parameters. Thanks to distributed computing techniques, it is possible today to routinely simulate local cortical networks of around 105 neurons with up to 109 synapses on clusters and multi-processor shared-memory machines. However, brain-scale networks are orders of magnitude larger than such local networks, in terms of numbers of neurons and synapses as well as in terms of computational load. Such networks have been investigated in individual studies, but the underlying simulation technologies have neither been described in sufficient detail to be reproducible nor made publicly available. Here, we discover that as the network model sizes approach the regime of meso- and macroscale simulations, memory consumption on individual compute nodes becomes a critical bottleneck. This is especially relevant on modern supercomputers such as the Blue Gene/P architecture where the available working memory per CPU core is rather limited. We develop a simple linear model to analyze the memory consumption of the constituent components of neuronal simulators as a function of network size and the number of cores used. This approach has multiple benefits. The model enables identification of key contributing components to memory saturation and prediction of the effects of potential improvements to code before any implementation takes place. As a consequence, development cycles can be shorter and less expensive. Applying the model to our freely available Neural Simulation Tool (NEST), we identify the software components dominant at different scales, and develop general strategies for reducing the memory consumption, in particular by using data structures that exploit the sparseness of the local representation of the network. We show that these adaptations enable our simulation software to scale up to the order of 10,000 processors and beyond. As memory consumption issues are likely to be relevant for any software dealing with complex connectome data on such architectures, our approach and our findings should be useful for researchers developing novel neuroinformatics solutions to the challenges posed by the connectome project.

A multi-scale modeling approach for studying cortical lesions as a cause for epilepsy

Traumatic brain injury (TBI) may result in post-traumatic seizures and epilepsy. Approximately 5-7% ofTBI patients suffer from at least one seizure [1]. Thepathophysiological mechanisms are not completelyunderstood, and may also differ between early seizures(< 2 weeks of the injury) and the development of post-traumatic epilepsy. This includes the effects of directphysical trauma, excitotoxicity due to iron released fromthe blood [2] and cytokine TGF-b in blood-brain-barrier-mediated activation of astrocytes [3]. In thiswork we propose that a reduction in network connectiv-ity, as presumed present in several cases of TBI, mayresult in seizures and epilepsy.We use a realistic model of neocortex consisting of sixdifferent, multi-compartmental neurons and Hodgkin-Huxley like ion-channel dynamics [4]. Using physiologi-cally realistic connectivityparameters, we analyze net-works of different sizes; ranging from a microcolumn of656 neurons to a mesocolumn that contains 20k neu-rons. In these networks, small lesions are introduced tosimulate axonal and dendritic damage, thereby limitingaction potential propagation. Furthermore, we analyze alumped model of neocortex that is shown to correspondto the detailed model of the microcolumn [5]. Thismodel consists of a system of two differential equationswith two fixed delays. By using an automated parameterestimation method, parameters are identified for whichthe model’s behavior closely resembles that of the realis-tic model. Subsequently, the dependency and sensitivityon these parameters are studied with bifurcation analy-sis. We generate a mesocolumn by linking severallumped units together. Lesions are then introduced bybreaking or reducing some of the connections betweenthe populations. We also study this case using bothanalytical and numerical bifurcation methods.The ratio between excitatory and inhibitory connec-tions is analytically determined as a function of networksize. It is found that, compared to large networks, smallnetworks tend to have a relatively larger number ofexcitatory connections than inhibitory connections. Thissuggests that a lesion splitting the network into smallersub-networks, could increase the ratio of excitatory andinhibitory?connections in a particular sub-network.Choosing parameters that correspond to a region ofmultistability, as determined by the bifurcation analysis,enables us to create an epileptic focus that spreadsepileptiform activity to neighboring areas.ConclusionsBy using multi-scale modeling, large-scale simulations,bifurcation analysis, and parameter estimation, we studythe effects of small lesions in neocortex. From the large-scale simulations we find that“neuronal peninsulas”,created by these lesions, may evolve into epileptogenicnetworks. By studying bifurcations of a lumped modelwith suitable parameters, regions of multistability areidentified that are hypothesized to correspond withepilepsy.

RuleMonkey: software for stochastic simulation of rule-based models

BackgroundThe system-level dynamics of many molecular interactions, particularly protein-protein interactions, can be conveniently represented using reaction rules, which can be specified using model-specification languages, such as the BioNetGen language (BNGL). A set of rules implicitly defines a (bio)chemical reaction network. The reaction network implied by a set of rules is often very large, and as a result, generation of the network implied by rules tends to be computationally expensive. Moreover, the cost of many commonly used methods for simulating network dynamics is a function of network size. Together these factors have limited application of the rule-based modeling approach. Recently, several methods for simulating rule-based models have been developed that avoid the expensive step of network generation. The cost of these "network-free" simulation methods is independent of the number of reactions implied by rules. Software implementing such methods is now needed for the simulation and analysis of rule-based models of biochemical systems.ResultsHere, we present a software tool called RuleMonkey, which implements a network-free method for simulation of rule-based models that is similar to Gillespie's method. The method is suitable for rule-based models that can be encoded in BNGL, including models with rules that have global application conditions, such as rules for intramolecular association reactions. In addition, the method is rejection free, unlike other network-free methods that introduce null events, i.e., steps in the simulation procedure that do not change the state of the reaction system being simulated. We verify that RuleMonkey produces correct simulation results, and we compare its performance against DYNSTOC, another BNGL-compliant tool for network-free simulation of rule-based models. We also compare RuleMonkey against problem-specific codes implementing network-free simulation methods.ConclusionsRuleMonkey enables the simulation of rule-based models for which the underlying reaction networks are large. It is typically faster than DYNSTOC for benchmark problems that we have examined. RuleMonkey is freely available as a stand-alone application http://public.tgen.org/rulemonkey. It is also available as a simulation engine within GetBonNie, a web-based environment for building, analyzing and sharing rule-based models.

An Analytically Tractable Model of Large Network

This paper considers specially organized networks of large size. They can serve as models of computer communication systems, economical systems, neural and genetic networks. The topology of this network is simple and the analysis of the network behaviour is an analytically tractable task, while computer simulations are difficult. The authors show that such networks generate any structurally stable attractors in particular chaotic and periodic. They can simulate all Turing machines, that is, perform any computations. In noisy cases, the reliability of such network is exponentially high as a function of network size and has a maximum for an optimal network size.