Leaky Integrator Neurons Research Articles

Memory in linear recurrent neural networks in continuous time

Reservoir Computing is a novel technique which employs recurrent neural networks while circumventing difficult training algorithms. A very recent trend in Reservoir Computing is the use of real physical dynamical systems as implementation platforms, rather than the customary digital emulations. Physical systems operate in continuous time, creating a fundamental difference with the classic discrete time definitions of Reservoir Computing. The specific goal of this paper is to study the memory properties of such systems, where we will limit ourselves to linear dynamics. We develop an analytical model which allows the calculation of the memory function for continuous time linear dynamical systems, which can be considered as networks of linear leaky integrator neurons. We then use this model to research memory properties for different types of reservoir. We start with random connection matrices with a shifted eigenvalue spectrum, which perform very poorly. Next, we transform two specific reservoir types, which are known to give good performance in discrete time, to the continuous time domain. Reservoirs based on uniform spreading of connection matrix eigenvalues on the unit disk in discrete time give much better memory properties than reservoirs with random connection matrices, where reservoirs based on orthogonal connection matrices in discrete time are very robust against noise and their memory properties can be tuned. The overall results found in this work yield important insights into how to design networks for continuous time.

Functional identification of biological neural networks using reservoir adaptation for point processes

The complexity of biological neural networks does not allow to directly relate their biophysical properties to the dynamics of their electrical activity. We present a reservoir computing approach for functionally identifying a biological neural network, i.e. for building an artificial system that is functionally equivalent to the reference biological network. Employing feed-forward and recurrent networks with fading memory, i.e. reservoirs, we propose a point process based learning algorithm to train the internal parameters of the reservoir and the connectivity between the reservoir and the memoryless readout neurons. Specifically, the model is an Echo State Network (ESN) with leaky integrator neurons, whose individual leakage time constants are also adapted. The proposed ESN algorithm learns a predictive model of stimulus-response relations in in vitro and simulated networks, i.e. it models their response dynamics. Receiver Operating Characteristic (ROC) curve analysis indicates that these ESNs can imitate the response signal of a reference biological network. Reservoir adaptation improved the performance of an ESN over readout-only training methods in many cases. This also held for adaptive feed-forward reservoirs, which had no recurrent dynamics. We demonstrate the predictive power of these ESNs on various tasks with cultured and simulated biological neural networks.

Electronic model of a dubois fuzzy integration neuron

In this work, we present a fuzzy electronic neuron that has a Dubois fuzzy integration method, an activation function with a fuzzy threshold, and a fuzzy response. We generated a fuzzy sum of the input signals and a shooting threshold value defined by means of a triangular or sinusoidal membership function. We present the electronic circuits, the oscilograms of the neuron responses, the value of the fuzzy integral, and we compare their behavior with those of a conventional leaky integrator neuron.

Simulating global properties of electroencephalograms with minimal random neural networks

The human electroencephalogram (EEG) is globally characterized by a 1 / f power spectrum superimposed with certain peaks, whereby the “alpha peak” in a frequency range of 8–14 Hz is the most prominent one for relaxed states of wakefulness. We present simulations of a minimal dynamical network model of leaky integrator neurons attached to the nodes of an evolving directed and weighted random graph (an Erdős–Rényi graph). We derive a model of the dendritic field potential (DFP) for the neurons leading to a simulated EEG that describes the global activity of the network. Depending on the network size, we find an oscillatory transition of the simulated EEG when the network reaches a critical connectivity. This transition, indicated by a suitably defined order parameter, is reflected by a sudden change of the network's topology when super-cycles are formed from merging isolated loops. After the oscillatory transition, the power spectra of simulated EEG time series exhibit a 1 / f continuum superimposed with certain peaks.

A model of biological motion perception from configural form cues.

Biological motion perception is the compelling ability of the visual system to perceive complex human movements effortlessly and within a fraction of a second. Recent neuroimaging and neurophysiological studies have revealed that the visual perception of biological motion activates a widespread network of brain areas. The superior temporal sulcus has a crucial role within this network. The roles of other areas are less clear. We present a computational model based on neurally plausible assumptions to elucidate the contributions of motion and form signals to biological motion perception and the computations in the underlying brain network. The model simulates receptive fields for images of the static human body, as found by neuroimaging studies, and temporally integrates their responses by leaky integrator neurons. The model reveals a high correlation to data obtained by neurophysiological, neuroimaging, and psychophysical studies.

Temporal Correlations in Stochastic Models of Double Bursting During Simulated Locomotion

The output of the spinal central pattern generator for locomotion falls into two broad categories: alternation between antagonistic muscles and double bursting within muscles acting on multiple joints. We first model an alternating half-center and then present two different models of double bursting. The first double-bursting model consists of a central clock with an explicit one-to-one mapping between interneuron activity and model output. The second double-bursting model consists of a half-center with an added feedback neuron. Models are built using rate-coded leaky integrator neurons with slow self-inhibition. Structure-function relationships are explored by the addition of noise. The interaction of noise with the dynamics of each network creates a unique pattern of correlation between phases of the simulated cycle. The effects of noise can be explained by perturbation of deterministic versions of the networks. Three basic results were obtained: slow self-inhibitory currents lead to correlations between parts of the step cycle that are separated in time and network relative; model outputs are most sensitive to perturbations presented just before competitive switches in network activity, and clock-like models possess substantial symmetries within the correlation structure of burst durations, whereas the correlation structure of feedback models are asymmetric. Our models suggest that variability in burst length durations can be analyzed to make inferences about the structure of the spinal networks for locomotion. In particular, correlation patterns within double-bursting outputs may yield important clues regarding the interaction between more central, clock-like networks and feedback from more peripheral interneurons.

Identification of prosodic attitudes by a temporal recurrent network

Human speakers modulate the fundamental frequency (F0) of their utterances in order to express different ‘prosodic’ attitudes such as surprise or curiosity. How are these prosodic attitudes then decoded? The current research addresses the issue of how the temporal structure of F0 can be used in order to discriminate between prosodic attitudes in natural language using a temporal recurrent neural network (TRN) that was initially developed to simulate the neurophysiology of the primate frontostriatal system. In the TRN, a recurrent network of leaky integrator neurons encodes a continuous trajectory of internal states that characterizes the input sequence. The input to the model is a population coding of the continuous, time-varying values of the fundamental frequency (F0) of natural language sentences. We expose the model to an experiment based on one in which human subjects were required to discriminate between different prosodic attitudes (surprise, exclamation, question, etc.). After training, the model discriminates between six prosodic attitudes in new sentences at 82.52% correct, compared to 72.8% correct for human subjects. These results reveal (1) that F0 provides relevant information for prosodic attitude discrimination, and (2) that the TRN demonstrates a categorical sensitivity to this information that can be used for classifying new sentences.

Spike timing dependent plasticity and mutual information in spiking neurons

Synaptic plasticity that delicately depends on the relative timing of pre- and post-synaptic spikes was recently observed in several brain areas. Here we derive a time-dependent learning rule from the basic principle of mutual information maximization for a leaky integrator neuron with spiking inputs, and study its relation to the experimentally observed rule. The comparison shows that not only is the biological learning rule similar in form to the analytically derived one, but it also increases mutual information to a near-optimal level. The analysis yields insights into the temporal characteristics of the observed learning rule and its dependency on neuronal biophysical parameters.

Analysis of neural spike trains with interspike interval reconstruction.

As a method for the analysis of neural spike trains, we examine fundamental characteristics of interspike interval (ISI) reconstruction theoretically with a leaky-integrator neuron model and experimentally with cricket wind receptor cells. Both the input to the leaky integrator and the stimulus to the wind receptor cells are the time series generated from the Rossler system. By numerical analysis of the leaky integrator, it is shown that, even if ISI reconstruction is possible, sometimes the entire structure of the Rössler attractor may not be reconstructed with ISI reconstruction. For analysis of the in vivo physiological responses of cricket wind receptor cells, we apply ISI reconstruction, nonlinear prediction and the surrogate data method to the experimental data. As a result of the analysis, it is found that there is a significant deterministic structure in the spike trains. By this analysis of physiological data, it is also shown that, even if ISI reconstruction is possible, the entire attractor may not be reconstructed.

Evolution of Neural Controllers for Locomotion and Obstacle Avoidance in a Six-legged Robot

This article describes how the SGOCE paradigm has been used within the context of a 'minimal simulation' strategy to evolve neural networks controlling locomotion and obstacle avoidance in a six-legged robot. A standard genetic algorithm has been used to evolve developmental programs according to which recurrent networks of leaky-integrator neurons were grown in a user-provided developmental substrate and were connected to the robot's sensors and actuators. Specific grammars have been used to limit the complexity of the developmental programs and of the corresponding neural controllers. Such controllers were first evolved through simulation and then successfully downloaded on the real robot.

An accelerator for neural networks with pulse-coded model neurons

The labeling of features by synchronization of spikes seems to be a very efficient encoding scheme for a visual system. Simulation of a vision system with millions of pulse-coded model neurons, however, is almost impossible on the base of available processors including parallel processors and neurocomputers. A "one-to-one" silicon implementation of pulse-coded model neurons suffers from communication problems and low flexibility. On the other hand, acceleration of the simulation algorithm of pulse-coded leaky integrator neurons has proved to be straightforward, flexible, and very efficient. Thus we decided to develop an accelerator for a special version of the French and Stein neurons with modulatory inputs which are advantageous for simulation of synchronization mechanisms. Moreover, our accelerator also provides a Hebbian-like learning rule and supports adaptivity. Up to 128 K neurons with a total number of 16 M freely allocatable synapses are simulated within one system. The size of networks, however, is not at all limited by these numbers as the system may be arbitrarily expanded. Simulation speed obviously depends on the number of interconnections and on the average activity within the network. In the case of locally interconnected networks for simulation of vision mechanisms there is only a very low percentage of simultaneously active neurons: stimuli are not simultaneously presented in all orientations and at all positions of the visual field. In these cases our accelerator provides close to real-time behavior if one second of a biological neuron is simulated by 1000 time slots.

RST: a connectionist architecture to deal with spatiotemporal relationships.

In the past decade, connectionism has proved its efficiency in the field of static pattern recognition. The next challenge is to deal with spatiotemporal problems. This article presents a new connectionist architecture, RST (réseau spatio temporel [spatio temporal network]), with such spatiotemporal capacities. It aims at taking into account at the architecture level both spatial relationships (e.g., as between neighboring pixels in an image) and temporal relationships (e.g., as between consecutive images in a video sequence). Concerning the spatial aspect, the network is embedded in actual space (two) or three-dimensional-, the metrics of which directly influence its structure through a connection distribution function. For the temporal aspect, we looked toward biology and used a leaky-integrator neuron model with a refractory period and postsynaptic potentials. The propagation of activity by spatiotemporal synchronized waves enables RST to perform motion detection and localization in sequences of video images.

And the beat goes on

The perception, manipulation and production of sequential events is a fundamental cognitive ability that plays a significant role in our linguistic and musical skills. While the order of the sequential events is a key component of this type of stimulus, the temporal dimension is also very important. For example, the performance of a serial reaction time (SRT) task was impaired by disruption of the order of the stimuli in the learned sequence or by disrupting the temporal organization of the learned sequence. A neural network approach described by Peter Dominey[ 1 Dominey P.F. A shared system for learning serial and temporal structure of sensori-motor sequences?: evidence from simulation and human experiments. Cognit. Brain. Res. 1988; 6: 163-172 Crossref Scopus (39) Google Scholar ]in a recent paper examined whether the temporal order and the serial order are represented within a single system. In the sequence learning model, a recurrent network of 5×5 leaky-integrator neurons were associated by reinforcement learning to the correct sequential responses. The neuronal elements in the recurrent network generated a succession of distinct patterns of activation during the sequence. Each pattern was then associated though learning with each of the corresponding responses in the succession of sequence trials. This resulted in error-free sequence reproduction and/or reduced reaction times as a function of the task. The model was then exposed to a standard SRT experiment in which reaction times for stimuli that were presented in repeating sequences were found to be reduced relative to sequences that were random—so providing evidence of the expected sequence learning. However, because the temporal structure of the delays between the stimuli were imbedded in the serial order of the sequences, the model was also sensitive to disruption of the temporal structure of the sequences. These results suggest that the serial and temporal structure of a sequence is encoded within a single system. The predictions derived from the neural simulations were subsequently confirmed in experimental studies with human subjects and might enhance our understanding of human sequence-learning.

A shared system for learning serial and temporal structure of sensori-motor sequences? Evidence from simulation and human experiments

This research investigates the influences of temporal structure on the representation of serial order. Experiments are performed in a neural network model of sequence learning and in human subjects. In the sequence learning model, a recurrent network of leaky integrator neurons encodes a succession of internal states that become associated, by reinforcement learning, with the correct sequential responses. First, the model is shown to learn a simple temporal discrimination task. The model is then exposed to two novel serial reaction time (SRT) experiments. In the standard SRT task (M.J. Nissen, P. Bullemer, Attentional requirements of learning: evidence from performance measures, Cogn. Psychol. 19 (1987) 1–32 [16]), reaction times for stimuli presented in a repeating sequence are reduced with respect to those for random stimuli, providing a measure of sequence learning. The novelty of the current experiments is that imbedded in the serial order of the sequences, there is a temporal structure of delays. The model is sensitive to both the serial structure and the temporal structure of the sequences. This observation is then confirmed in human subjects. These results demonstrate how a novel recurrent architecture encodes the interaction of temporal and serial structure and provide insight into related aspects of human sensori-motor sequence learning.

Role of Temporal Integration and Fluctuation Detection in the Highly Irregular Firing of a Leaky Integrator Neuron Model with Partial Reset

Partial reset is a simple and powerful tool for controlling the irregularity of spike trains fired by a leaky integrator neuron model with random inputs. In particular, a single neuron model with a realistic membrane time constant of 10 ms can reproduce the highly irregular firing of cortical neurons reported by Softky and Koch (1993). In this article, the mechanisms by which partial reset affects the firing pattern are investigated. Itisshown theoretically that partial reset is equivalent to the use of a time-dependent threshold, similar to a technique proposed by Wilbur and Rinzel (1983) to produce high irregularity. This equivalent model allows establishing that temporal integration and fluctuation detection can coexist and cooperate to cause highly irregular firing. This study also reveals that reverse correlation curves cannot be used reliably to assess the causes of firing. For instance, they do not reveal temporal integration when it takes place. Further, the peak near time zero does not always indicate coincidence detection. An alternative qualitative method is proposed here for that later purpose. Finally, it is noted that as the reset becomes weaker, the firing pattern shows a progressive transition from regular firing, to random, to temporally clustered, and eventually to bursting firing. Concurrently the slope of the transfer function increases. Thus, simulations suggest a correlation between high gain and highly irregular firing.

L-2 Clinical applications of functional MRI

Categorical encoding is crucial for mastering large bodies of related sensory-motor experiences, but what is its neural substrate? In an effort to respond to this question, recent single-unit recording studies in the macaque lateral prefrontal cortex (LPFC) have demonstrated two characteristic forms of neural encoding of the sequential structure of the animal’s sensory-motor experience. One population of neurons encodes the specific behavioral sequences. A second population of neurons encodes the sequence category (e.g. ABAB, AABB or AAAA) and does not differentiate sequences within the category (Shima, K., Isoda, M., Mushiake, H., Tanji, J., 2007. Categorization of behavioural sequences in the prefrontal cortex. Nature 445, 315–318.). Interestingly these neurons are intermingled in the lateral prefrontal cortex, and not topographically segregated. Thus, LPFC may provide a neurophysiological basis for sensorimotor categorization. Here we report on a neural network simulation study that reproduces and explains these results. We model a cortical circuit composed of three layers (infragranular, granular, and supragranular) of 5 * 5 leaky integrator neurons with a sigmoidal output function, and we examine 1000 such circuits running in parallel. Crucially the three layers are interconnected with recurrent connections, thus producing a dynamical system that is inherently sensitive to the spatiotemporal structure of the sequential inputs. The model is presented with 11 four-element sequences following Shima et al. We isolated one subpopulation of neurons each of whose activity predicts individual sequences, and a second population that predicts category independent of the specific sequence. We argue that a richly interconnected cortical circuit is capable of internally generating a neural representation of category membership, thus significantly extending the scope of recurrent network computation. In order to demonstrate that these representations can be used to create an explicit categorization capability, we introduced an additional neural structure corresponding to the striatum. We showed that via cortico-striatal plasticity, neurons in the striatum could produce an explicit representation both of the identity of each sequence, and its category membership.

NEURAL NETWORK CAPACITY FOR TEMPORAL SEQUENCE STORAGE

Gardner’s analysis of the capacity of a neural net for pattern storage is extended to the case of the storage of temporal sequences (TSS). This is done for leaky integrator neurons of the simplest form, and the dependence of the capacity on basin size determined.

Storing temporal sequences

Direct storage of temporal sequences is analysed in terms of a neural net composed of leaky integrator neurons with a range of time constants. These neurons store previously presented patterns and allow the transitions between the patterns of a sequence to be learnt. This is shown even to lead to disambiguation (which is defined in Section 1). Storage capacity is determined by simulation. We also present a detailed study of the efficiency of this system in its dependence on the type of neuronal activity. Finally, we note the relevance of our model to understanding activity in the hippocampus.

Temporal summation in rat prefrontal pyramidal cells. Differential effects of pre- and postsynaptic neurochemical manipulations

The influence of both reserpine-induced depletion of catecholamines and chlorpromazine-induced blockade of dopamine receptors on integrative properties of rats prefrontal pyramidal cells was studied by evocating threshold direct cortical responses with different trains of electrical pulses. Catecholamine depletion results in increased pulse train stimulating currents for eliciting threshold cortical responses, whereas chlorpromazine blockade of dopamine receptors results in higher time constants characterizing postsynaptic temporal summation. It is suggested that the leaky-integrator neuron model may be a discriminating paradigm for detecting electrophysiologically the pre- or postsynaptic level of occurrence of some synaptic disorders.

Input-output relationship of the Leaky-integrator neuron model.

This paper presents a method of calculating the spike sequence at the output of the Leaky-Integrator Neuron Model (LIM) in response to an arbitrary input stimulus. The calculations have revealed new properties of the initial transient behavior of the LIM, as well as new constraints upon necessary and sufficient conditions for the appearance of spikes with a fixed phase relation to a periodic input. It is also possible to infer what knowledge about the input stimulus can be obtained from a temporal sequence of spikes at the output of the LIM. In the Discussion, neuronal examples are considered which do not encode the transmitted information as a spike rate, but rather monitor the time of occurence of individual spikes by comparison with a reference signal. This relatively common case is not adequately treated by previous descriptions based on system theory; ways are suggested by which the formalism developed here can be used to describe completely, and understand more fully, the performance of such systems.