Attractor Neural Networks Research Articles

Bayesian sampling in visual perception

It is well-established that some aspects of perception and action can be understood as probabilistic inferences over underlying probability distributions. In some situations, it would be advantageous for the nervous system to sample interpretations from a probability distribution rather than commit to a particular interpretation. In this study, we asked whether visual percepts correspond to samples from the probability distribution over image interpretations, a form of sampling that we refer to as Bayesian sampling. To test this idea, we manipulated pairs of sensory cues in a bistable display consisting of two superimposed moving drifting gratings, and we asked subjects to report their perceived changes in depth ordering. We report that the fractions of dominance of each percept follow the multiplicative rule predicted by Bayesian sampling. Furthermore, we show that attractor neural networks can sample probability distributions if input currents add linearly and encode probability distributions with probabilistic population codes.

Enhancing neural-network performance via assortativity

The performance of attractor neural networks has been shown to depend crucially on the heterogeneity of the underlying topology. We take this analysis a step further by examining the effect of degree-degree correlations--assortativity--on neural-network behavior. We make use of a method recently put forward for studying correlated networks and dynamics thereon, both analytically and computationally, which is independent of how the topology may have evolved. We show how the robustness to noise is greatly enhanced in assortative (positively correlated) neural networks, especially if it is the hub neurons that store the information.

Learning sequences of sparse correlated patterns using small-world attractor neural networks: An application to traffic videos

The goal of this work is to learn and retrieve a sequence of highly correlated patterns using a Hopfield-type of attractor neural network (ANN) with a small-world connectivity distribution. For this model, we propose a weight learning heuristic which combines the pseudo-inverse approach with a row-shifting schema. The influence of the ratio of random connectivity on retrieval quality and learning time has been studied. Our approach has been successfully tested on a complex pattern, as it is the case of traffic video sequences, for different combinations of the involved parameters. Moreover, it has demonstrated to be robust with respect to highly variable frame activity.

Memory Dynamics in Attractor Networks with Saliency Weights

Memory is a fundamental part of computational systems like the human brain. Theoretical models identify memories as attractors of neural network activity patterns based on the theory that attractor (recurrent) neural networks are able to capture some crucial characteristics of memory, such as encoding, storage, retrieval, and long-term and working memory. In such networks, long-term storage of the memory patterns is enabled by synaptic strengths that are adjusted according to some activity-dependent plasticity mechanisms (of which the most widely recognized is the Hebbian rule) such that the attractors of the network dynamics represent the stored memories. Most of previous studies on associative memory are focused on Hopfield-like binary networks, and the learned patterns are often assumed to be uncorrelated in a way that minimal interactions between memories are facilitated. In this letter, we restrict our attention to a more biological plausible attractor network model and study the neuronal representations of correlated patterns. We have examined the role of saliency weights in memory dynamics. Our results demonstrate that the retrieval process of the memorized patterns is characterized by the saliency distribution, which affects the landscape of the attractors. We have established the conditions that the network state converges to unique memory and multiple memories. The analytical result also holds for other cases for variable coding levels and nonbinary levels, indicating a general property emerging from correlated memories. Our results confirmed the advantage of computing with graded-response neurons over binary neurons (i.e., reducing of spurious states). It was also found that the nonuniform saliency distribution can contribute to disappearance of spurious states when they exit.

Stability Analysis of Attractor Neural Network Model of Inferior Temporal Cortex –Relationship between Attractor Stability and Learning Order–

Miyashita found that the long-term memory of visual stimuli is stored in the monkey's inferior temporal cortex and that the temporal correlation in terms of the learning order of visual stimuli is converted into spatial correlation in terms of the firing rate patterns of the neuron group. To explain Miyashita's findings, Griniasty et al. [Neural Comput. 5 (1993) 1] and Amit et al. [J. Neurosci. 14 (1994) 6435] proposed the attractor neural network model, and the Amit model has been examined only for the stable state acquired by storing memory patterns in a fixed sequence. In the real world, however, the learning order has statistical continuity but it also has randomness, and the stability of the state changes depending on the statistical properties of learning order when memory patterns are stored randomly. In addition, it is preferable for the stable state to become an appropriate attractor that reflects the relationship between memory patterns by the statistical properties of the learning order. In thi...

Bistable, Irregular Firing and Population Oscillations in a Modular Attractor Memory Network

Attractor neural networks are thought to underlie working memory functions in the cerebral cortex. Several such models have been proposed that successfully reproduce firing properties of neurons recorded from monkeys performing working memory tasks. However, the regular temporal structure of spike trains in these models is often incompatible with experimental data. Here, we show that the in vivo observations of bistable activity with irregular firing at the single cell level can be achieved in a large-scale network model with a modular structure in terms of several connected hypercolumns. Despite high irregularity of individual spike trains, the model shows population oscillations in the beta and gamma band in ground and active states, respectively. Irregular firing typically emerges in a high-conductance regime of balanced excitation and inhibition. Population oscillations can produce such a regime, but in previous models only a non-coding ground state was oscillatory. Due to the modular structure of our network, the oscillatory and irregular firing was maintained also in the active state without fine-tuning. Our model provides a novel mechanistic view of how irregular firing emerges in cortical populations as they go from beta to gamma oscillations during memory retrieval.

Annotations for symmetric probabilistic encryption algorithm based on chaotic attractors of neural networks

The security of the symmetric probabilistic encryption scheme based on chaotic attractors of neural networks is analyzed and discussed. Firstly, the key uniqueness is proved by analyzing the rotation transform matrix to avoid the attack of the equivalent key. Secondly, the distributed uniformity of the numbers “0” and “1” in the corresponding attracting domain for every chaotic attractor is analyzed by the statistics method. It is testified that the distributed uniformity can be kept if the synaptic matrix of the neural network is changed by a standard permutation matrix. Two annotations based on the results above are proposed to improve the application security of the encryption algorithm.

Internal representation of task rules by recurrent dynamics: the importance of the diversity of neural responses

Neural activity of behaving animals, especially in the prefrontal cortex, is highly heterogeneous, with selective responses to diverse aspects of the executed task. We propose a general model of recurrent neural networks that perform complex rule-based tasks, and we show that the diversity of neuronal responses plays a fundamental role when the behavioral responses are context-dependent. Specifically, we found that when the inner mental states encoding the task rules are represented by stable patterns of neural activity (attractors of the neural dynamics), the neurons must be selective for combinations of sensory stimuli and inner mental states. Such mixed selectivity is easily obtained by neurons that connect with random synaptic strengths both to the recurrent network and to neurons encoding sensory inputs. The number of randomly connected neurons needed to solve a task is on average only three times as large as the number of neurons needed in a network designed ad hoc. Moreover, the number of needed neurons grows only linearly with the number of task-relevant events and mental states, provided that each neuron responds to a large proportion of events (dense/distributed coding). A biologically realistic implementation of the model captures several aspects of the activity recorded from monkeys performing context-dependent tasks. Our findings explain the importance of the diversity of neural responses and provide us with simple and general principles for designing attractor neural networks that perform complex computation.

Statistical mechanics of attractor neural network models with synaptic depression

Synaptic depression is known to control gain for presynaptic inputs. Since cortical neurons receive thousands of presynaptic inputs, and their outputs are fed into thousands of other neurons, the synaptic depression should influence macroscopic properties of neural networks. We employ simple neural network models to explore the macroscopic effects of synaptic depression. Systems with the synaptic depression cannot be analyzed due to asymmetry of connections with the conventional equilibrium statistical-mechanical approach. Thus, we first propose a microscopic dynamical mean field theory. Next, we derive macroscopic steady state equations and discuss the stabilities of steady states for various types of neural network models.

Tracking dynamics of two-dimensional continuous attractor neural networks

We introduce an analytically solvable model of two-dimensional continuous attractor neural networks (CANNs). The synaptic input and the neuronal response form Gaussian bumps in the absence of external stimuli, and enable the network to track external stimuli by its translational displacement in the two-dimensional space. Basis functions of the two-dimensional quantum harmonic oscillator in polar coordinates are introduced to describe the distortion modes of the Gaussian bump. The perturbative method is applied to analyze its dynamics. Testing the method by considering the network behavior when the external stimulus abruptly changes its position, we obtain results of the reaction time and the amplitudes of various distortion modes, with excellent agreement with simulation results.

Nontrivial Global Attractors in 2-D Multistable Attractor Neural Networks

Attractor dynamics is a crucial problem for attractor neural networks, as it is the underling computational mechanism for memory storage and retrieval in neural systems. This brief studies a class of attractor network consisting of linearized threshold neurons, and analyzes global attractors based on a parameterized 2-D model. On the basis of previous results on nondegenerate and degenerate equilibria in mathematics, we further elucidate all possible nontrivial global attractors. Our theoretical result provides precise descriptions on how the changes of network parameters affect the attractors' distribution and landscape, and it may give a feasible solution towards specifying attractors by specifying weights. Simulations are presented to illustrate the theoretical results.

Symmetric sequence processing in a recurrent neural network model with a synchronous dynamics

The synchronous dynamics and the stationary states of a recurrent attractor neural network model with competing synapses between symmetric sequence processing and Hebbian pattern reconstruction are studied in this work allowing for the presence of a self-interaction for each unit. Phase diagrams of stationary states are obtained exhibiting phases of retrieval, symmetric and period-two cyclic states as well as correlated and frozen-in states, in the absence of noise. The frozen-in states are destabilized by synaptic noise and well-separated regions of correlated and cyclic states are obtained. Excitatory or inhibitory self-interactions yield enlarged phases of fixed-point or cyclic behaviour.

Neural basis of perceptual expectations: insights from transient dynamics of attractor neural networks

Sensory information from the external world is intrinsically ambiguous, necessitating prior experience as a constraint on perception to parse stimuli into well-defined categories. Priming, which have been used vastly to study such perceptual influences, is the phenomenon where the perception of a given stimulus, or the prime, affects the perception of a succeeding stimulus, or the target, even when the target is presented after a long delay or the prime is not explicitly perceived [1]. Using classical priming paradigms, it is observed that brief exposure to a stimulus – ranging from tens to hundreds of milliseconds – biases subjects to perceive the following stimuli either as dissimilar (adaptation aftereffects) or more similar (priming) to the priming stimuli [2,3]. Thus, in a categorization task, the category boundary may move towards or away from the prime. The duration of the prime and the prime-target asynchrony, as well as the type of intervening mask, affect both the strength and the direction of the effect [2,4]. Current evidence has led to the suggestion that a particular type of the observed priming effect may be primarily due to the conflicting acts of lingering activation from the prime and accumulating depletion of synaptic resources [5]. Some other studies looked at the interaction between local shunting adaptation and a near-threshold neural baseline [6]. This neural model explains the observed behavior, without invoking any high-level decision making or memory process. However, it has been shown that also high-level, complex processes such as face perception are subject to aftereffects, namely rate firing adaptation. In spite of a large body of empirical data on various behavioral outcomes in such experiments, we still lack an explanatory model capable of predicting the crossover from adaptation aftereffect to priming, as emerging in a generic cortical network. We have developed an analytical approach to study the transient dynamics of networks of threshold-linear model neurons that include, as a necessary ingredient of the relevant computational mechanism, a simple feature of pyramidal cell biophysics: firing rate adaptation. The analysis yields the attractor states of the network and the full spectrum of time constants of the transients associated with different steady states. Studying these transients, in the response to external inputs that are morphed between two stored patterns, and affected by previous activity of the network, could shed light on the possible contribution of attractor dynamics to perceptual boundary shifts. The preliminary results show that firing rate adaptation plays the main role to produce adaptation aftereffects in our network, without which one only observes priming effects, if any. The relative duration of target length and adaptation time scale is one of the crucial terms determining the dynamics. The strength of recurrent connection with respect to feedforward inputs is another relevant parameter.

Block attractor in spatially organized neural networks

The model of attractor neural network on the small-world topology (local and random connectivity) is investigated. The synaptic weights are random, driving the network towards a disordered state for the neural activity. An ordered macroscopic neural state is induced by a bias in the network weight connections, and the network evolution when initialized in blocks of positive/negative activity is studied. The retrieval of the block-like structure is investigated. An application to the Hebbian learning of a pattern, carrying local information, is presented. The block and the global attractor compete according to the initial conditions and the change of stability from one to the other depends on the long-range character of the network connectivity, as shown with a flow-diagram analysis. Moreover, a larger number of blocks emerges with the network dilution.

POLYCHRONOUS WAVEFRONT COMPUTATIONS

There is great interest in methods for computing that do not involve digital machines. Many computational paradigms were inspired by brain research, such as Boolean neuronal logic [McCulloch & Pitts, 1943], the perceptron [Rosenblatt, 1958], attractor neural networks [Hopfield, 1982] and cellular neural nets [Chua & Yang, 1988]. All these paradigms abstract biological circuits to artificial neural networks, i.e. interconnected units (neurons) that perform computations based on the connections between the units (synapses). Here we present a novel computational framework based on polychronous wavefront dynamics. It is entirely different from an artificial neural network paradigm, rather it is based on temporal and spatial patterns of activity in pulse-propagating media and their interaction with transponders, which create pulses in response to receiving appropriate inputs, e.g. two coincident input pulses. A pulse propagates as a circular wave from its source to other transponders. Computations result from interactions between transponders, and they are encoded by the exact physical locations of transponders and by precise timings of pulses. We illustrate temporal pattern recognition, reverberating memory, temporal signal analysis and basic logical operations using polychronous wavefront computations. This work reveals novel principles for designing nanoscale computational devices.

A Moving Bump in a Continuous Manifold: A Comprehensive Study of the Tracking Dynamics of Continuous Attractor Neural Networks

Understanding how the dynamics of a neural network is shaped by the network structure and, consequently, how the network structure facilitates the functions implemented by the neural system is at the core of using mathematical models to elucidate brain functions. This study investigates the tracking dynamics of continuous attractor neural networks (CANNs). Due to the translational invariance of neuronal recurrent interactions, CANNs can hold a continuous family of stationary states. They form a continuous manifold in which the neural system is neutrally stable. We systematically explore how this property facilitates the tracking performance of a CANN, which is believed to have clear correspondence with brain functions. By using the wave functions of the quantum harmonic oscillator as the basis, we demonstrate how the dynamics of a CANN is decomposed into different motion modes, corresponding to distortions in the amplitude, position, width, or skewness of the network state. We then develop a perturbation approach that utilizes the dominating movement of the network's stationary states in the state space. This method allows us to approximate the network dynamics up to an arbitrary accuracy depending on the order of perturbation used. We quantify the distortions of a gaussian bump during tracking and study their effects on tracking performance. Results are obtained on the maximum speed for a moving stimulus to be trackable and the reaction time for the network to catch up with an abrupt change in the stimulus.

CHAOS IN HETEROGENEOUS NETWORKS WITH TEMPORALLY INERT NODES

We discuss an attractor neural network in which only a fraction ρ of nodes is simultaneously updated. In addition, the network has a heterogeneous distribution of connection weights and, depending on the current degree of order, connections are changed at random by a factor Φ on short-time scales. The resulting dynamic attractors may become unstable in a certain range of Φ thus ensuing chaotic itineracy which highly depends on ρ. For intermediate values of ρ, we observe that the number of attractors visited increases with ρ, and that the trajectory may change from regular to chaotic and vice versa as ρ is modified. Statistical analysis of time series shows a power-law spectra under conditions in which the attractors' space is most efficiently explored.

Attractor dynamics in VLSI

Event Abstract Back to Event Attractor dynamics in VLSI Patrick Camilleri1*, Massimiliano Giulioni2, Maurizio Mattia2, Jochen Braun1 and Paolo Del Giudice2 1 Otto-von-Guericke University, Germany 2 Italian National Institute of Health, Italy We describe and demonstrate the implementation of attractor neural network dynamics in analog VLSI technology on the F-LANN chip [1]. The on-chip network is made up of an excitatory and an inhibitory population consisting of 128 linear integrate-and-fire neurons recurrently connected together. Apart from the recurrent input these two populations receive external input in the form of Poisson distributed spike trains from an Address-Event-Representation (AER) based system. These external stimuli are needed to provide an actual stimulus to the attractor network as well as to provide an adequate 'thermal-bath' for the on-chip populations. We explain how by starting from a theoretical mean-field approximation of a hypothetical attractor neural network having two stable states of activity, we progress to find the correct chip parameters (voltage biases) in order to obtain an on-chip effective response function (EFR) that matches with the theoretical EFR [2]. Once this is achieved we proceed to demonstrate that the hardware attractor neural network really shows the attractor behavior by having a spontaneous state and a working memory state. The measured attractor activity matches favorably with the mean-field and software simulation results.

Multiple Stability of a Sparsely Encoded Attractor Neural Network Model for the Inferior Temporal Cortex

We study a neural network model for the inferior temporal cortex, in terms of finite memory loading and sparse coding. We show that an uncorrelated Hopfield-type attractor and some correlated attractors have multiple stability, and examine the retrieval dynamics for these attractors when the initial state is set to a noise-degraded memory pattern. Then, we show that there is a critical initial overlap: that is, the system converges to the correlated attractor when the noise level is large, and otherwise to the Hopfield-type attractor. Furthermore, we study the time course of the correlation between the correlated attractors in the retrieval dynamics. On the basis of these theoretical results, we resolve the controversy regarding previous physiologic experimental findings regarding neuron properties in the inferior temporal cortex and propose a new experimental paradigm.

Running memory span: A comparison of behavioral capacity limits with those of an attractor neural network

We studied a computational model of short term memory capacity that performs a simulated running memory span task using Hebbian learning and rapid decay of connection strengths to keep recent items active for later recall. This model demonstrates recall performance similar to humans performing the same task, with a capacity limit of approximately three items and a prominent recency effect. The model also shows that this capacity depends on decay to release the model from accumulating interference. Model findings are compared with data from two behavioral experiments that used varying task demands to tax memory capacity limits. Following additional theoretical predictions from the computational model, behavioral data support that when task demands require attention to be spread too thin to keep items available for later recall, capacity limits suffer. These findings are important both for understanding the mechanisms underlying short term memory capacity, and also to memory researchers interested in the role of attention in capacity limitations.