Random Recurrent Neural Networks Research Articles

Discontinuous transition to chaos in a canonical random neural network.

We study a paradigmatic random recurrent neural network introduced by Sompolinsky, Crisanti, and Sommers (SCS). In the infinite size limit, this system exhibits a direct transition from a homogeneous rest state to chaotic behavior, with the Lyapunov exponent gradually increasing from zero. We generalize the SCS model considering odd saturating nonlinear transfer functions, beyond the usual choice ϕ(x)=tanhx. A discontinuous transition to chaos occurs whenever the slope of ϕ at 0 is a local minimum [i.e., for ϕ^{'''}(0)>0]. Chaos appears out of the blue, by an attractor-repeller fold. Accordingly, the Lyapunov exponent stays away from zero at the birth of chaos.

Random recurrent neural networks with delays

An infinite lattice model of a recurrent neural network with random connection strengths between neurons is developed and analyzed. To incorporate the presence of various type of delays in the neural networks, both discrete and distributed time varying delays are considered in the model. For the existence of random pullback attractors and periodic attractors, the nonlinear terms of the resulting system are not expected to be Lipschitz continuous, but only satisfy a weaker continuity assumption along with growth conditions, under which the uniqueness of the underlying Cauchy problem may not hold. Then after extending the concept and theory of monotone multi-valued semiflows to the random context, the structure of random pullback attractors with or without periodicity is investigated. In particular, the existence and stability properties of extremal random complete trajectories are studied.

Probabilistic vs deep learning based approaches for narrow domain NER in Spanish

This work presents an experimental study on the task of Named Entity Recognition (NER) for a narrow domain in Spanish language. This study considers two approaches commonly used in this kind of problem, namely, a Conditional Random Fields (CRF) model and Recurrent Neural Network (RNN). For the latter, we employed a bidirectional Long Short-Term Memory with ELMO’s pre-trained word embeddings for Spanish. The comparison between the probabilistic model and the deep learning model was carried out in two collections, the Spanish dataset from CoNLL-2002 considering four classes under the IOB tagging schema, and a Mexican Spanish news dataset with seventeen classes under IOBES schema. The paper presents an analysis about the scalability, robustness, and common errors of both models. This analysis indicates in general that the BiLSTM-ELMo model is more suitable than the CRF model when there is “enough” training data, and also that it is more scalable, as its performance was not significantly affected in the incremental experiments (by adding one class at a time). On the other hand, results indicate that the CRF model is more adequate for scenarios having small training datasets and many classes.

Independent Random Recurrent Neural Networks for Infrared Spatial Point Targets Classification

Exo-atmospheric infrared (IR) point target discrimination is an important research topic of space surveillance systems. It is difficult to describe the characteristic information of the shape and micro-motion states of the targets and to discriminate different targets effectively by the characteristic information. This paper has constructed the infrared signature model of spatial point targets and obtained the infrared radiation intensity sequences dataset of different types of targets. This paper aims to design an algorithm for the classification problem of infrared radiation intensity sequences of spatial point targets. Recurrent neural networks (RNNs) are widely used in time series classification tasks, but face several problems such as gradient vanishing and explosion, etc. In view of shortcomings of RNNs, this paper proposes an independent random recurrent neural network (IRRNN) model, which combines independent structure RNNs with randomly weighted RNNs. Without increasing the training complexity of network learning, our model solves the problem of gradient vanishing and explosion, improves the ability to process long sequences, and enhances the comprehensive classification performance of the algorithm effectively. Experiments show that the IRRNN algorithm performs well in classification tasks and is robust to noise.

Takens-inspired neuromorphic processor: A downsizing tool for random recurrent neural networks via feature extraction

We describe a new technique which minimizes the amount of neurons in the hidden layer of a random recurrent neural network (rRNN) for time series prediction. Merging Takens-based attractor reconstruction methods with machine learning, we identify a mechanism for feature extraction that can be leveraged to lower the network size. We obtain criteria specific to the particular prediction task and derive the scaling law of the prediction error. The consequences of our theory are demonstrated by designing a Takens-inspired hybrid processor, which extends a rRNN with a priori designed delay external memory. Our hybrid architecture is therefore designed including both, real and virtual nodes. Via this symbiosis, we show performance of the hybrid processor by stabilizing an arrhythmic neural model. Thanks to our obtained design rules, we can reduce the stabilizing neural network's size by a factor of 15 with respect to a standard system.

Random Neural Networks with Hierarchical Committees for Improved Routing in Wireless Mesh Networks with Interference

We propose a hierarchical (nested) variant of a recurrent random neural network (RNN) with reinforced learning, introduced by Gelenbe. Each neuron (committee) in a top-level RNN represents a different bottom-level RNN (or sub-committee). The bottom-level RNNs choose the best routing and the top-level RNN chooses the currently best bottom-level RNN. Each of the bottom RNNs is trained in a different way. When they differ in their choice of the best path, several cognitive packets are routed according to the different decisions. In that case, a respective ACK packet trains individual bottom RNNs and not all bottom RNNs at once. An example presents an optimisation of a real-time routing in a dense mesh network of wireless sensors relaying small metering messages between each other, until the messages reach a common gateway. The network is experiencing a periodic electromagnetic interference. The hierarchical variant causes a small increase in the number of smart packets but allows a considerably better routing quality.

A novel method for structure selection of the Recurrent Random Neural Network using multiobjective optimisation

The Random Neural Network (RNN) has extensively investigated over the past few decades; this research has resulted in a considerable number of theoretical and application papers. Although, great effort has been done to develop a systematic procedure to train the recurrent fashion of the RNN, the choice of the number of neurons remains an open question. To overcome this problem, at least partially, this paper uses multiobjective optimisation (MOP) to select the number of neurons. The MOP framework used the mean square error (MSE) and the number of neurons (N) as the objectives to be minimised. The stochastic nondominated algorithm (SNA) to exclude dominated solutions of the Pareto-set has been also introduced. Instead of using only the best solution, candidates to the Pareto-set are excluded by statistical comparison among mean values of the two objectives in all training runs. The SNA allows a statistically correct exclusion of dominated solutions; the best solution can be picked up using classical decision-making procedures. Numerical and real examples illustrate the potentiality of the proposed method in two areas: classification problems and system identification.

Reconstruct Recurrent Neural Networks via Flexible Sub-Models for Time Series Classification

Recurrent neural networks (RNNs) remain challenging, and there is still a lack of long-term memory or learning ability in sequential data classification and prediction. In this paper, we propose a flexible recurrent model, BIdirectional COnvolutional RaNdom RNNs (BICORN-RNNs), incorporating a series of sub-models: random projection, convolutional operation, and bidirectional transmission. These subcategories advance classification accuracy, which was limited by the gradient vanishing and the exploding problem. Experiments on public time series datasets demonstrate that our proposed method substantially outperforms a variety of existing models. Furthermore, the coordination of the accuracy and efficiency concerning a variety of factors, including SNR, length, data missing, and overlapping, is also discussed.

Sensory Stream Adaptation in Chaotic Networks

Implicit expectations induced by predictable stimuli sequences affect neuronal response to upcoming stimuli at both single cell and neural population levels. Temporally regular sensory streams also phase entrain ongoing low frequency brain oscillations but how and why this happens is unknown. Here we investigate how random recurrent neural networks without plasticity respond to stimuli streams containing oddballs. We found the neuronal correlates of sensory stream adaptation emerge if networks generate chaotic oscillations which can be phase entrained by stimulus streams. The resultant activity patterns are close to critical and support history dependent response on long timescales. Because critical network entrainment is a slow process stimulus response adapts gradually over multiple repetitions. Repeated stimuli generate suppressed responses but oddball responses are large and distinct. Oscillatory mismatch responses persist in population activity for long periods after stimulus offset while individual cell mismatch responses are strongly phasic. These effects are weakened in temporally irregular sensory streams. Thus we show that network phase entrainment provides a biologically plausible mechanism for neural oddball detection. Our results do not depend on specific network characteristics, are consistent with experimental studies and may be relevant for multiple pathologies demonstrating altered mismatch processing such as schizophrenia and depression.

Low-dimensional dynamics of structured random networks.

Using a generalized random recurrent neural network model, and by extending our recently developed mean-field approach [J. Aljadeff, M. Stern, and T. Sharpee, Phys. Rev. Lett. 114, 088101 (2015)], we study the relationship between the network connectivity structure and its low-dimensional dynamics. Each connection in the network is a random number with mean 0 and variance that depends on pre- and postsynaptic neurons through a sufficiently smooth function g of their identities. We find that these networks undergo a phase transition from a silent to a chaotic state at a critical point we derive as a function of g. Above the critical point, although unit activation levels are chaotic, their autocorrelation functions are restricted to a low-dimensional subspace. This provides a direct link between the network's structure and some of its functional characteristics. We discuss example applications of the general results to neuroscience where we derive the support of the spectrum of connectivity matrices with heterogeneous and possibly correlated degree distributions, and to ecology where we study the stability of the cascade model for food web structure.

A local Echo State Property through the largest Lyapunov exponent

Echo State Networks are efficient time-series predictors, which highly depend on the value of the spectral radius of the reservoir connectivity matrix. Based on recent results on the mean field theory of driven random recurrent neural networks, enabling the computation of the largest Lyapunov exponent of an ESN, we develop a cheap algorithm to establish a local and operational version of the Echo State Property.

Recurrent networks expect

How the brain anticipates future events is one of the most intriguing questions in neural information processing. The brain does not simply respond to the external world but in predictably structured environments like natural language, birdsong and somatosensory flow in fluids predicts it [1]. Reactions are improved by predictive encoding of what sensory stimuli may occur as well as when they may occur. Quantitative studies of implicit timing in streaming perceptual discrimination tasks show that performance is enhanced if stimuli occur at expected times according to an established rhythm. The neural basis for implicit timing is not fully understood but oscillatory entrainment mechanisms have been suggested [2]. Recent studies confirm that low-frequency brain oscillations do become phase entrained in such tasks. Phase entrainment increases with the temporal regularity of the sensory stream and correlates with enhanced discrimination performance [3]. It is not limited to sensory cortices; larger scale cortical networks, as well as subcortical networks like the basal ganglia [4], may also be coherently modulated by the predictability of stimuli streams. The presence of large scale oscillatory entrainment suggests that network mediated neuronal ensemble dynamics may be involved. Recurrent neural networks generate complex but reproducible temporally extended dynamical activity patterns in response to input stimuli [5,6]. Such transient activity patterns have been suggested to provide a natural substrate for working memory and a representation of elapsed time. Here we add to the understanding of how random recurrent neural networks support neural information processing by demonstrating that temporal expectation also naturally emerges from their dynamics. We show that the weakly chaotic oscillations generated by recurrent networks can be phase synchronized [7] by temporally regular stimulus sequences. We show network responses are maximally discriminative when stimuli fall at their preferred phase in phase entrained networks (Figure ​(Figure1).1). Discriminability increases continuously with both temporal regularity and stimulus type predictability which also interact. Our results do not depend on specific network characteristics, are resilient to the presence of network noise and random distractor stimuli and are consistent with multiple streaming perceptual discrimination studies. Figure 1 Excess correct classification probability of on-time target stimuli compared to mistimed target stimuli (100 ms early, black ; 100 ms late, red) in a temporally regular streaming task with inter-trial-interval 800 ms. Results are plotted versus the Lyapunov ...

A critical study of network models for neural networks and their dynamics

We use three network models, Erdős–Rényi, Watts–Strogatz and structured nodes, to generate networks sharing several topological features with the neural network of C. elegans (our target network). A new topological measurement, incoming and outgoing edges heat maps, is introduced and used to compare the considered networks. We run these networks as random recurrent neural networks and study their dynamics.We find out that none of the considered network models generates networks similar to the target one both in their topological features and dynamics. Moreover, we find that the dynamics of the target network are very robust to the rewiring of its edges.

Oscillation induced propagation of synchrony in structured neural networks

Spatio-temporally coordinated patterns of spiking activity have been experimentally observed in a range of neural circuits, but their dynamical origin is still not well understood. A prominent hypothesis states that propagating synchronized activity through embedded feed-forward networks might dynamically generate such patterns [1,2]. Modeling studies indeed showed that such synfire-chains embedded in random recurrent networks enable reliable signal transmission by propagating localized (sub-network) synchrony, if their structure is strongly pronounced compared to the embedding network. This requires in particular a dense connectivity between the neuronal layers of the chain or strongly enhanced synapses and modified response properties of neurons within the chain [3]. So far, however, such prominent large-scale structures have not been experimentally observed. Single neuron experiments [4] indicate that neuronal dendrites are capable of nonlinearly amplifying sufficiently synchronous inputs by eliciting dendritic spikes, thereby inducing non-additive interactions. Here we demonstrate that such non-additive coupling promotes guided synchrony propagation already in random recurrent neural networks with mildly enhanced, biologically plausible sub-structures and without anatomically superimposed feed-forward chains [5]. Our analysis explains the mechanisms underlying robust propagation and shows in which sense non-additive enhancement -- a local neuron property that dynamically changes with input synchrony -- may complement dense and non-local structural connectivity. Most neuronal circuits exhibit oscillations of various frequencies and amplitudes [6]. Such oscillations may influence the dynamics of synchrony propagation. We thus further study this influence for both externally induced oscillations as well as for oscillations generated by the rhythmically propagating synchronous activity itself. We find that in networks with linear dendrites and balanced input, the oscillations hinder synchrony propagation, if they effect the dynamics at all. In contrast, for non-additive coupling, oscillations support synchronous propagation, if they are in resonance and the interplay between oscillations and propagating activity induces complex locking patterns: We show that in recurrent circuits containing high-connectivity (hub-)neurons, the oscillatory echo to propagating synchrony can generate synchrony in the remainder of the network and thereby in turn stabilize or even enable synchrony propagation along predefined paths: The network echo promotes signal transmission.

Guiding Synchrony through Random Networks

Sparse random networks contain structures that can be considered as diluted feed-forward networks. Modeling of cortical circuits has shown that feed-forward structures, if strongly pronounced compared to the embedding random network, enable reliable signal transmission by propagating localized (sub-network) synchrony. This assumed prominence, however, is not experimentally observed in local cortical circuits. Here we show that nonlinear dendritic interactions as discovered in recent single neuron experiments, naturally enable guided synchrony propagation already in random recurrent neural networks exhibiting mildly enhanced, biologically plausible sub-structures.

Recurrent Kernel Machines: Computing with Infinite Echo State Networks

Echo state networks (ESNs) are large, random recurrent neural networks with a single trained linear readout layer. Despite the untrained nature of the recurrent weights, they are capable of performing universal computations on temporal input data, which makes them interesting for both theoretical research and practical applications. The key to their success lies in the fact that the network computes a broad set of nonlinear, spatiotemporal mappings of the input data, on which linear regression or classification can easily be performed. One could consider the reservoir as a spatiotemporal kernel, in which the mapping to a high-dimensional space is computed explicitly. In this letter, we build on this idea and extend the concept of ESNs to infinite-sized recurrent neural networks, which can be considered recursive kernels that subsequently can be used to create recursive support vector machines. We present the theoretical framework, provide several practical examples of recursive kernels, and apply them to typical temporal tasks.

A Mathematical Analysis of the Effects of Hebbian Learning Rules on the Dynamics and Structure of Discrete-Time Random Recurrent Neural Networks

We present a mathematical analysis of the effects of Hebbian learning in random recurrent neural networks, with a generic Hebbian learning rule, including passive forgetting and different timescales, for neuronal activity and learning dynamics. Previous numerical work has reported that Hebbian learning drives the system from chaos to a steady state through a sequence of bifurcations. Here, we interpret these results mathematically and show that these effects, involving a complex coupling between neuronal dynamics and synaptic graph structure, can be analyzed using Jacobian matrices, which introduce both a structural and a dynamical point of view on neural network evolution. Furthermore, we show that sensitivity to a learned pattern is maximal when the largest Lyapunov exponent is close to 0. We discuss how neural networks may take advantage of this regime of high functional interest.

Random recurrent neural networks dynamics

This paper is a review dealing with the study of large size random recurrent neural networks. The connection weights are varying according to a probability law and it is possible to predict the network dynamics at a macroscopic scale using an averaging principle. After a first introductory section, the section 2 reviews the various models from the points of view of the single neuron dynamics and of the global network dynamics. A summary of notations is presented, which is quite helpful for the sequel. In section 3, mean-field dynamics is developed. The probability distribution characterizing global dynamics is computed. In section 4, some applications of mean-field theory to the prediction of chaotic regime for Analog Formal Random Recurrent Neural Networks (AFRRNN) are displayed. The case of AFRRNN with an homogeneous population of neurons is studied in section 4.1. Then, a two-population model is studied in section 4.2. The occurrence of a cyclo-stationary chaos is displayed using the results of [16]. In section 5, an insight of the application of mean-field theory to IF networks is given using the results of [9].

Learning and control with large dynamic neural networks

This paper is a presentation of neuronal control systems in the terms of the dynamical systems theory, where (1) the controller and its surrounding environment are seen as two co-dependent controlled dynamical systems (2) the behavioral transitions that take place under adaptation processes are analyzed in terms of phase-transitions. We present in the second section a generic method for the construction of multi-population random recurrent neural networks. The third section gives an overview of the various phase transitions that take place under an external forcing signal, or under internal parametric changes. The section 4 presents some applications in the domain of sequence identification and active perception modeling. The section 5 presents some applications in the domain of closed-loop control systems and reinforcement learning.

Effects of Hebbian learning on the dynamics and structure of random networks with inhibitory and excitatory neurons

The aim of the present paper is to study the effects of Hebbian learning in random recurrent neural networks with biological connectivity, i.e. sparse connections and separate populations of excitatory and inhibitory neurons. We furthermore consider that the neuron dynamics may occur at a (shorter) time scale than synaptic plasticity and consider the possibility of learning rules with passive forgetting. We show that the application of such Hebbian learning leads to drastic changes in the network dynamics and structure. In particular, the learning rule contracts the norm of the weight matrix and yields a rapid decay of the dynamics complexity and entropy. In other words, the network is rewired by Hebbian learning into a new synaptic structure that emerges with learning on the basis of the correlations that progressively build up between neurons. We also observe that, within this emerging structure, the strongest synapses organize as a small-world network. The second effect of the decay of the weight matrix spectral radius consists in a rapid contraction of the spectral radius of the Jacobian matrix. This drives the system through the “edge of chaos” where sensitivity to the input pattern is maximal. Taken together, this scenario is remarkably predicted by theoretical arguments derived from dynamical systems and graph theory.