Linear Recurrent Networks Research Articles

Memory and forecasting capacities of nonlinear recurrent networks

The notion of memory capacity, originally introduced for echo state and linear networks with independent inputs, is generalized to nonlinear recurrent networks with stationary but dependent inputs. The presence of dependence in the inputs makes natural the introduction of the network forecasting capacity, that measures the possibility of forecasting time series values using network states. Generic bounds for memory and forecasting capacities are formulated in terms of the number of neurons of the nonlinear recurrent network and the autocovariance function or the spectral density of the input. These bounds generalize well-known estimates in the literature to a dependent inputs setup. Finally, for the particular case of linear recurrent networks with independent inputs it is proved that the memory capacity is given by the rank of the associated controllability matrix, a fact that has been for a long time assumed to be true without proof by the community.

Generation of Scale-Invariant Sequential Activity in Linear Recurrent Networks.

Sequential neural activity has been observed in many parts of the brain and has been proposed as a neural mechanism for memory. The natural world expresses temporal relationships at a wide range of scales. Because we cannot know the relevant scales a priori, it is desirable that memory, and thus the generated sequences, is scale invariant. Although recurrent neural network models have been proposed as a mechanism for generating sequences, the requirements for scale-invariant sequences are not known. This letter reports the constraints that enable a linear recurrent neural network model to generate scale-invariant sequential activity. A straightforward eigendecomposition analysis results in two independent conditions that are required for scale invariance for connectivity matrices with real, distinct eigenvalues. First, the eigenvalues of the network must be geometrically spaced. Second, the eigenvectors must be related to one another via translation. These constraints are easily generalizable for matrices that have complex and distinct eigenvalues. Analogous albeit less compact constraints hold for matrices with degenerate eigenvalues. These constraints, along with considerations on initial conditions, provide a general recipe to build linear recurrent neural networks that support scale-invariant sequential activity.

Publisher's Note: Difference between memory and prediction in linear recurrent networks [Phys. Rev. E 96 , 032308 (2017)

Received 23 October 2018DOI:https://doi.org/10.1103/PhysRevE.98.049902©2018 American Physical SocietyPhysics Subject Headings (PhySH)Research AreasComputational complexityMathematical physicsScaling laws of complex systemsInterdisciplinary Physics

Difference between memory and prediction in linear recurrent networks.

Recurrent networks are trained to memorize their input better, often in the hopes that such training will increase the ability of the network to predict. We show that networks designed to memorize input can be arbitrarily bad at prediction. We also find, for several types of inputs, that one-node networks optimized for prediction are nearly at upper bounds on predictive capacity given by Wiener filters and are roughly equivalent in performance to randomly generated five-node networks. Our results suggest that maximizing memory capacity leads to very different networks than maximizing predictive capacity and that optimizing recurrent weights can decrease reservoir size by half an order of magnitude.

Integration as sequence detection in a feedforward neural integrator

In neural integrator networks, transient inputs are accumulated into sustained output signals that reflect the mathematical integral, over time, of their inputs. This computation has been identified as an important component of a wide variety of brain functions ranging from accumulation of sensory evidence for decision making to the motor control of eye movements. All current network models of neural integration assume that the conversion of transient inputs to sustained responses is accomplished by feedback among recurrently connected neuronal elements. Here we show that neural integration can occur even in feedforward networks and describe the properties of this novel class of integrators. We consider a feedforward network consisting of multiple stages that each have a time constant τ with which they linearly filter their inputs. We show that the effective dynamics of this network can be reduced to that of a simple network consisting of a linear chain of neurons with input entering one end and getting successively filtered by each successive stage of the network. As a result of this filtering, later stages of the network have prolonged responses that peak at successively later times. Thus, the network effectively forms a delay-line set of basis functions that are localized in time and that can be flexibly summed to generate a variety of temporal responses. We show analytically that with appropriate choices of synaptic weights, the network can perform a nearly perfect integral of its inputs over a duration of time of order Nτ, where N is the number of stages in the network. We further show that although the performance of the network is best understood in terms of basis functions corresponding to a delay-line, the responses of the actual neurons in the network will generally be linear combinations of these basis functions that may not be easily recognized as originating from dynamics governed by a delay line. The robustness of the network to uniform changes in all synaptic weights can be shown analytically to be similar to that of linear recurrent networks, exhibiting exponential decay of the integrated activity if the weights are too small and exponential growth until signals begin to exit the network if the weights are too large. We show that proper tuning of the weights can be accomplished by a homeostatic learning rule in which neurons scale their intrinsic gain and/or synaptic weights until their activity reaches an average target level over time. In conclusion, this work suggests a novel mechanism for neural integration. Although we focus on its role as an integrator, the network bears strong similarities to previous networks proposed for temporal sequence recognition and production. This suggests that common underlying principles may be relevant to a host of temporal processing computations.

Short-Term Memory in Orthogonal Neural Networks

We study the ability of linear recurrent networks obeying discrete time dynamics to store long temporal sequences that are retrievable from the instantaneous state of the network. We calculate this temporal memory capacity for both distributed shift register and random orthogonal connectivity matrices. We show that the memory capacity of these networks scales with system size.