Dynamic Recurrent Network Research Articles

Dynamic fading memory and expectancy effects in the monkey primary visual cortex

In order to investigate the involvement of the primary visual cortex (V1) in working memory (WM), parallel, multisite recordings of multi-unit activity were obtained from monkey V1 while the animals performed a delayed match-to-sample (DMS) task. During the delay period, V1 population firing rate vectors maintained a lingering trace of the sample stimulus that could be reactivated by intervening impulse stimuli that enhanced neuronal firing. This fading trace of the sample did not require active engagement of the monkeys in the DMS task and likely reflects the intrinsic dynamics of recurrent cortical networks in lower visual areas. This renders an active, attention-dependent involvement of V1 in the maintenance of WM contents unlikely. By contrast, population responses to the test stimulus depended on the probabilistic contingencies between sample and test stimuli. Responses to tests that matched expectations were reduced which agrees with concepts of predictive coding.

Graph convolutional dynamic recurrent network with attention for traffic forecasting

Graph convolutional dynamic recurrent network with attention for traffic forecasting

TMS-evoked responses are driven by recurrent large-scale network dynamics.

A compelling way to disentangle the complexity of the brain is to measure the effects of spatially and temporally synchronized systematic perturbations. In humans, this can be non-invasively achieved by combining transcranial magnetic stimulation (TMS) and electroencephalography (EEG). Spatiotemporally complex and long-lasting TMS-EEG evoked potential (TEP) waveforms are believed to result from recurrent, re-entrant activity that propagates broadly across multiple cortical and subcortical regions, dispersing from and later re-converging on, the primary stimulation site. However, if we loosely understand the TEP of a TMS-stimulated region as the impulse response function of a noisy underdamped harmonic oscillator, then multiple later activity components (waveform peaks) should be expected even for an isolated network node in the complete absence of recurrent inputs. Thus emerges a critically important question for basic and clinical research on human brain dynamics: what parts of the TEP are due to purely local dynamics, what parts are due to reverberant, re-entrant network activity, and how can we distinguish between the two? To disentangle this, we used source-localized TMS-EEG analyses and whole-brain connectome-based computational modelling. Results indicated that recurrent network feedback begins to drive TEP responses from 100 ms post-stimulation, with earlier TEP components being attributable to local reverberatory activity within the stimulated region. Subject-specific estimation of neurophysiological parameters additionally indicated an important role for inhibitory GABAergic neural populations in scaling cortical excitability levels, as reflected in TEP waveform characteristics. The novel discoveries and new software technologies introduced here should be of broad utility in basic and clinical neuroscience research.

Relating local connectivity and global dynamics in recurrent excitatory-inhibitory networks.

How the connectivity of cortical networks determines the neural dynamics and the resulting computations is one of the key questions in neuroscience. Previous works have pursued two complementary approaches to quantify the structure in connectivity. One approach starts from the perspective of biological experiments where only the local statistics of connectivity motifs between small groups of neurons are accessible. Another approach is based instead on the perspective of artificial neural networks where the global connectivity matrix is known, and in particular its low-rank structure can be used to determine the resulting low-dimensional dynamics. A direct relationship between these two approaches is however currently missing. Specifically, it remains to be clarified how local connectivity statistics and the global low-rank connectivity structure are inter-related and shape the low-dimensional activity. To bridge this gap, here we develop a method for mapping local connectivity statistics onto an approximate global low-rank structure. Our method rests on approximating the global connectivity matrix using dominant eigenvectors, which we compute using perturbation theory for random matrices. We demonstrate that multi-population networks defined from local connectivity statistics for which the central limit theorem holds can be approximated by low-rank connectivity with Gaussian-mixture statistics. We specifically apply this method to excitatory-inhibitory networks with reciprocal motifs, and show that it yields reliable predictions for both the low-dimensional dynamics, and statistics of population activity. Importantly, it analytically accounts for the activity heterogeneity of individual neurons in specific realizations of local connectivity. Altogether, our approach allows us to disentangle the effects of mean connectivity and reciprocal motifs on the global recurrent feedback, and provides an intuitive picture of how local connectivity shapes global network dynamics.

Estimation of a battery electric vehicle output power and remaining driving range under subfreezing conditions

This manuscript proposes a hybrid machine learning approach to estimate the battery output power and remaining driving range in a battery electric vehicle (BEV). These two metrics are among the most significant factors in the market penetration of BEVs, specifically in countries with harsh winters. The proposed hybrid method comprises a recurrent dynamic network, nonlinear autoregressive with exogenous inputs (NARX), and a recursive algorithm, Kalman filter (KF). The NARX neural network utilizes the information from the weather, the vehicle's speed, and road conditions to predict the battery output power, and the remaining driving range. KF, unlike the other similar studies, is employed to account for the dynamic changes in different road conditions by providing the vehicle rolling resistance and aerodynamic drag coefficient estimates to the network. To validate the performance of the proposed method, several driving tests under different ambient conditions are conducted using a real BEV (Kia Soul 2017), and the performance of the suggested approach is compared with a conventional model-based method. The comparison between the measured battery output power and the estimated one shows that the proposed hybrid method provides more accurate estimates than a model-based approach with over 50 % of percentage change in terms of root mean squared error. Furthermore, through a systemic network optimization, it is shown that the data from two trips (out of 14 trips) are sufficient to successfully predict the real-time battery output power, energy consumption, and the remaining driving range in different weather conditions, including winter. Therefore, the proposed NARX network is particularly suitable to circumvent the range anxiety issues and hence predict the remaining range better than the traditional vehicle dynamics-based approach.

The impact of sparsity in low-rank recurrent neural networks.

Neural population dynamics are often highly coordinated, allowing task-related computations to be understood as neural trajectories through low-dimensional subspaces. How the network connectivity and input structure give rise to such activity can be investigated with the aid of low-rank recurrent neural networks, a recently-developed class of computational models which offer a rich theoretical framework linking the underlying connectivity structure to emergent low-dimensional dynamics. This framework has so far relied on the assumption of all-to-all connectivity, yet cortical networks are known to be highly sparse. Here we investigate the dynamics of low-rank recurrent networks in which the connections are randomly sparsified, which makes the network connectivity formally full-rank. We first analyse the impact of sparsity on the eigenvalue spectrum of low-rank connectivity matrices, and use this to examine the implications for the dynamics. We find that in the presence of sparsity, the eigenspectra in the complex plane consist of a continuous bulk and isolated outliers, a form analogous to the eigenspectra of connectivity matrices composed of a low-rank and a full-rank random component. This analogy allows us to characterise distinct dynamical regimes of the sparsified low-rank network as a function of key network parameters. Altogether, we find that the low-dimensional dynamics induced by low-rank connectivity structure are preserved even at high levels of sparsity, and can therefore support rich and robust computations even in networks sparsified to a biologically-realistic extent.

Dynamical Characteristics of Recurrent Neuronal Networks Are Robust Against Low Synaptic Weight Resolution.

The representation of the natural-density, heterogeneous connectivity of neuronal network models at relevant spatial scales remains a challenge for Computational Neuroscience and Neuromorphic Computing. In particular, the memory demands imposed by the vast number of synapses in brain-scale network simulations constitute a major obstacle. Limiting the number resolution of synaptic weights appears to be a natural strategy to reduce memory and compute load. In this study, we investigate the effects of a limited synaptic-weight resolution on the dynamics of recurrent spiking neuronal networks resembling local cortical circuits and develop strategies for minimizing deviations from the dynamics of networks with high-resolution synaptic weights. We mimic the effect of a limited synaptic weight resolution by replacing normally distributed synaptic weights with weights drawn from a discrete distribution, and compare the resulting statistics characterizing firing rates, spike-train irregularity, and correlation coefficients with the reference solution. We show that a naive discretization of synaptic weights generally leads to a distortion of the spike-train statistics. If the weights are discretized such that the mean and the variance of the total synaptic input currents are preserved, the firing statistics remain unaffected for the types of networks considered in this study. For networks with sufficiently heterogeneous in-degrees, the firing statistics can be preserved even if all synaptic weights are replaced by the mean of the weight distribution. We conclude that even for simple networks with non-plastic neurons and synapses, a discretization of synaptic weights can lead to substantial deviations in the firing statistics unless the discretization is performed with care and guided by a rigorous validation process. For the network model used in this study, the synaptic weights can be replaced by low-resolution weights without affecting its macroscopic dynamical characteristics, thereby saving substantial amounts of memory.

From synapse to network: models of information storage and retrieval in neural circuits.

The mechanisms of information storage and retrieval in brain circuits are still the subject of debate. It is widely believed that information is stored at least in part through changes in synaptic connectivity in networks that encode this informationand that these changes lead in turn to modifications of network dynamics, such that the stored information can be retrieved at a later time. Here, we review recent progress in deriving synaptic plasticity rules from experimental dataand in understanding how plasticity rules affect the dynamics of recurrent networks. We show that the dynamics generated by such networks exhibit a large degree of diversity, depending on parameters, similar to experimental observations invivo during delayed response tasks.

Recurrent dynamics in the cerebral cortex: Integration of sensory evidence with stored knowledge

Current concepts of sensory processing in the cerebral cortex emphasize serial extraction and recombination of features in hierarchically structured feed-forward networks in order to capture the relations among the components of perceptual objects. These concepts are implemented in convolutional deep learning networks and have been validated by the astounding similarities between the functional properties of artificial systems and their natural counterparts. However, cortical architectures also display an abundance of recurrent coupling within and between the layers of the processing hierarchy. This massive recurrence gives rise to highly complex dynamics whose putative function is poorly understood. Here a concept is proposed that assigns specific functions to the dynamics of cortical networks and combines, in a unifying approach, the respective advantages of recurrent and feed-forward processing. It is proposed that the priors about regularities of the world are stored in the weight distributions of feed-forward and recurrent connections and that the high-dimensional, dynamic space provided by recurrent interactions is exploited for computations. These comprise the ultrafast matching of sensory evidence with the priors covertly represented in the correlation structure of spontaneous activity and the context-dependent grouping of feature constellations characterizing natural objects. The concept posits that information is encoded not only in the discharge frequency of neurons but also in the precise timing relations among the discharges. Results of experiments designed to test the predictions derived from this concept support the hypothesis that cerebral cortex exploits the high-dimensional recurrent dynamics for computations serving predictive coding.

An intelligent parametric modeling and identification of a 5 kW ballard PEM fuel cell system based on dynamic recurrent networks with delayed context units

This article presents a dynamic simulation study for the modeling and identification of a 5 kW Proton Exchange Membrane (PEM) fuel cell system using intelligent ANN approach to get rid of the complexity involved in the analytical modeling as it is intricate with the highly non-linear dynamics such as electrochemical, thermodynamic and water-transport mechanisms. The proposed artificial networks for the prediction and identification of a highly non-linear fuel cell system performance are Radial Basis Function Network (RBFN), dynamic Elman Recurrent Network (ERN) and NARX Recurrent Network (NRN) that has delayed context unit. A comparative study is made between the performances of the proposed neural network models for identifying an optimal network structure and configuration based on network performance measures. The optimal NARX network ascertained having an appreciable learning and generalization ability is adopted for the prediction and identification of the fuel cell system for its static and dynamic behavior. The network prediction over the system behavior shows good agreement with the benchmark results acquired from a 5 kW-Ballard fuel cell system. The proposed optimal intelligent parametric model can facilitate in replacing a highly non-linear fuel cell system in the fuel cell related model developments that are adopted in several research sectors particularly in transportation applications.

Recurrent network dynamics shape direction selectivity in primary auditory cortex

Detecting the direction of frequency modulation (FM) is essential for vocal communication in both animals and humans. Direction-selective firing of neurons in the primary auditory cortex (A1) has been classically attributed to temporal offsets between feedforward excitatory and inhibitory inputs. However, it remains unclear how cortical recurrent circuitry contributes to this computation. Here, we used two-photon calcium imaging and whole-cell recordings in awake mice to demonstrate that direction selectivity is not caused by temporal offsets between synaptic currents, but by an asymmetry in total synaptic charge between preferred and non-preferred directions. Inactivation of cortical somatostatin-expressing interneurons (SOM cells) reduced direction selectivity, revealing its cortical contribution. Our theoretical models showed that charge asymmetry arises due to broad spatial topography of SOM cell-mediated inhibition which regulates signal amplification in strongly recurrent circuitry. Together, our findings reveal a major contribution of recurrent network dynamics in shaping cortical tuning to behaviorally relevant complex sounds.

Network-Centered Homeostasis Through Inhibition Maintains Hippocampal Spatial Map and Cortical Circuit Function

Despite ongoing experiential change, neural activity maintains remarkable stability. Such stability is thought to be mediated by homeostatic plasticity and is deemed to be critical for normal neural function. However, what aspect of neural activity does homeostatic plasticity conserve, and how it still maintains the flexibility necessary for learning and memory, is not fully understood. Homeostatic plasticity is often studied in the context of neuron-centered control, where the deviations from the target activity for each individual neuron are suppressed. However, experimental studies suggest that there are additional, network-centered mechanisms. These may act through the inhibitory neurons, due to their dense network connectivity. Here we use a computational framework to study a potential mechanism for such homeostasis, using experimentally inspired, input-dependent inhibitory plasticity (IDIP). In a hippocampal CA1 spiking model, we show that IDIP in combination with place tuned input can explain the formation of active and silent place cells, as well as place cells remapping following optogenetic silencing of active place cells. Furthermore, we show that IDIP can also stabilise recurrent network dynamics, as well as preserve network firing rate heterogeneity and stimulus representation. Interestingly, in an associative memory task, IDIP facilitates persistent activity after memory encoding, in line with some experimental data. Hence, the establishment of global network balance with IDIP has diverse functional implications and may be able to explain experimental phenomena across different brain areas. Finally, we make testable experimental predictions.

An Artificial Neural Network Processor With a Custom Instruction Set Architecture for Embedded Applications

This article presents the design and implementation of an embedded programmable processor with a custom instruction set architecture for efficient realization of artificial neural networks (ANNs). The ANN processor architecture is scalable, supporting an arbitrary number of layers and number of artificial neurons (ANs) per layer. Moreover, the processor supports ANNs with arbitrary interconnect structures among ANs to realize both feed-forward and dynamic recurrent networks. The processor architecture is customizable in which the numerical representation of inputs, outputs, and signals among ANs can be parameterized to an arbitrary fixed-point format. An ASIC implementation of the designed programmable ANN processor for networks with up to 512 ANs and 262,000 interconnects is presented and is estimated to occupy 2.23 mm2 of silicon area and consume 1.25 mW of power from a 1.6 V supply while operating at 74 MHz in a standard 32-nm CMOS technology. In order to assess and compare the efficiency of the designed ANN processor, we have designed and implemented a dedicated reconfigurable hardware architecture for the direct realization of ANNs. Characteristics and implementation results of the designed programmable ANN processor and the dedicated ANN hardware on a Xilinx Artix-7 field-programmable gate array (FPGA) are presented and compared using two benchmarks, the MNIST benchmark using a feed-forward ANN and a movie review sentiment analysis benchmark using a recurrent neural network.

Dynamics of random recurrent networks with correlated low-rank structure

A given neural network in the brain is involved in many different tasks. This implies that, when considering a specific task, the network's connectivity contains a component which is related to the task and another component which can be considered random. Understanding the interplay between the structured and random components, and their effect on network dynamics and functionality is an important open question. Recent studies addressed the co-existence of random and structured connectivity, but considered the two parts to be uncorrelated. This constraint limits the dynamics and leaves the random connectivity non-functional. Algorithms that train networks to perform specific tasks typically generate correlations between structure and random connectivity. Here we study nonlinear networks with correlated structured and random components, assuming the structure to have a low rank. We develop an analytic framework to establish the precise effect of the correlations on the eigenvalue spectrum of the joint connectivity. We find that the spectrum consists of a bulk and multiple outliers, whose location is predicted by our theory. Using mean-field theory, we show that these outliers directly determine both the fixed points of the system and their stability. Taken together, our analysis elucidates how correlations allow structured and random connectivity to synergistically extend the range of computations available to networks.

Short-Term Attractive Tilt Aftereffects Predicted by a Recurrent Network Model of Primary Visual Cortex.

Adaptation is a multi-faceted phenomenon that is of interest in terms of both its function and its potential to reveal underlying neural processing. Many behavioral studies have shown that after exposure to an oriented adapter the perceived orientation of a subsequent test is repulsed away from the orientation of the adapter. This is the well-known Tilt Aftereffect (TAE). Recently, we showed that the dynamics of recurrently connected networks may contribute substantially to the neural changes induced by adaptation, especially on short time scales. Here we extended the network model and made the novel behavioral prediction that the TAE should be attractive, not repulsive, on a time scale of a few 100 ms. Our experiments, using a novel adaptation protocol that specifically targeted adaptation on a short time scale, confirmed this prediction. These results support our hypothesis that recurrent network dynamics may contribute to short-term adaptation. More broadly, they show that understanding the neural processing of visual inputs that change on the time scale of a typical fixation requires a detailed analysis of not only the intrinsic properties of neurons, but also the slow and complex dynamics that emerge from their recurrent connectivity. We argue that this is but one example of how even simple recurrent networks can underlie surprisingly complex information processing, and are involved in rudimentary forms of memory, spatio-temporal integration, and signal amplification.

Cortical reliability amid noise and chaos

Typical responses of cortical neurons to identical sensory stimuli appear highly variable. It has thus been proposed that the cortex primarily uses a rate code. However, other studies have argued for spike-time coding under certain conditions. The potential role of spike-time coding is directly limited by the internally generated variability of cortical circuits, which remains largely unexplored. Here, we quantify this internally generated variability using a biophysical model of rat neocortical microcircuitry with biologically realistic noise sources. We find that stochastic neurotransmitter release is a critical component of internally generated variability, causing rapidly diverging, chaotic recurrent network dynamics. Surprisingly, the same nonlinear recurrent network dynamics can transiently overcome the chaos in response to weak feed-forward thalamocortical inputs, and support reliable spike times with millisecond precision. Our model shows that the noisy and chaotic network dynamics of recurrent cortical microcircuitry are compatible with stimulus-evoked, millisecond spike-time reliability, resolving a long-standing debate.

Weight statistics controls dynamics in recurrent neural networks.

Recurrent neural networks are complex non-linear systems, capable of ongoing activity in the absence of driving inputs. The dynamical properties of these systems, in particular their long-time attractor states, are determined on the microscopic level by the connection strengths wij between the individual neurons. However, little is known to which extent network dynamics is tunable on a more coarse-grained level by the statistical features of the weight matrix. In this work, we investigate the dynamics of recurrent networks of Boltzmann neurons. In particular we study the impact of three statistical parameters: density (the fraction of non-zero connections), balance (the ratio of excitatory to inhibitory connections), and symmetry (the fraction of neuron pairs with wij = wji). By computing a ‘phase diagram’ of network dynamics, we find that balance is the essential control parameter: Its gradual increase from negative to positive values drives the system from oscillatory behavior into a chaotic regime, and eventually into stationary fixed points. Only directly at the border of the chaotic regime do the neural networks display rich but regular dynamics, thus enabling actual information processing. These results suggest that the brain, too, is fine-tuned to the ‘edge of chaos’ by assuring a proper balance between excitatory and inhibitory neural connections.

Contrasting the effects of adaptation and synaptic filtering on the timescales of dynamics in recurrent networks.

Neural activity in awake behaving animals exhibits a vast range of timescales that can be several fold larger than the membrane time constant of individual neurons. Two types of mechanisms have been proposed to explain this conundrum. One possibility is that large timescales are generated by a network mechanism based on positive feedback, but this hypothesis requires fine-tuning of the strength or structure of the synaptic connections. A second possibility is that large timescales in the neural dynamics are inherited from large timescales of underlying biophysical processes, two prominent candidates being intrinsic adaptive ionic currents and synaptic transmission. How the timescales of adaptation or synaptic transmission influence the timescale of the network dynamics has however not been fully explored. To address this question, here we analyze large networks of randomly connected excitatory and inhibitory units with additional degrees of freedom that correspond to adaptation or synaptic filtering. We determine the fixed points of the systems, their stability to perturbations and the corresponding dynamical timescales. Furthermore, we apply dynamical mean field theory to study the temporal statistics of the activity in the fluctuating regime, and examine how the adaptation and synaptic timescales transfer from individual units to the whole population. Our overarching finding is that synaptic filtering and adaptation in single neurons have very different effects at the network level. Unexpectedly, the macroscopic network dynamics do not inherit the large timescale present in adaptive currents. In contrast, the timescales of network activity increase proportionally to the time constant of the synaptic filter. Altogether, our study demonstrates that the timescales of different biophysical processes have different effects on the network level, so that the slow processes within individual neurons do not necessarily induce slow activity in large recurrent neural networks.

Adaptive Stimulus Design for Dynamic Recurrent Neural Network Models.

We present an adaptive stimulus design method for efficiently estimating the parameters of a dynamic recurrent network model with interacting excitatory and inhibitory neuronal populations. Although stimuli that are optimized for model parameter estimation should, in theory, have advantages over nonadaptive random stimuli, in practice it remains unclear in what way and to what extent an optimal design of time-varying stimuli may actually improve parameter estimation for this common type of recurrent network models. Here we specified the time course of each stimulus by a Fourier series whose amplitudes and phases were determined by maximizing a utility function based on the Fisher information matrix. To facilitate the optimization process, we have derived differential equations that govern the time evolution of the gradients of the utility function with respect to the stimulus parameters. The network parameters were estimated by maximum likelihood from the spike train data generated by an inhomogeneous Poisson process from the continuous network state. The adaptive design process was repeated in a closed loop, alternating between optimal stimulus design and parameter estimation from the updated stimulus-response data. Our results confirmed that, compared with random stimuli, optimally designed stimuli elicited responses with significantly better likelihood values for parameter estimation. Furthermore, all individual parameters, including the time constants and the connection weights, were recovered more accurately by the optimal design method. We also examined how the errors of different parameter estimates were correlated, and proposed heuristic formulas to account for the correlation patterns by an approximate parameter-confounding theory. Our results suggest that although adaptive optimal stimulus design incurs considerable computational cost even for the simplest excitatory-inhibitory recurrent network model, it may potentially help save time in experiments by reducing the number of stimuli needed for network parameter estimation.

A conceptual alternative forecasting model for alternative investments

In this article we present a conceptual model for forecasting purposes that can be used from fund managers or investors. Our conceptual model is a hybrid model and borrows concepts from machine learning; more specifically, from artificial neural networks (ANN) and fuzzy logic (FL). We propose the use of the nonlinear autoregressive network with exogenous inputs (NARX) which is a recurrent dynamic network, with feedback connections enclosing several layers of the network. This ANN is combined with a FL component to deal with uncertainties when considering various market conditions. The proposed conceptual forecasting model has an open architecture design so as to be extended and optimized based on investors’ needs.