Dynamics Of Associative Memory Research Articles

SSTE: Syllable-Specific Temporal Encoding to FORCE-learn audio sequences with an associative memory approach

The circuitry and pathways in the brains of humans and other species have long inspired researchers and system designers to develop accurate and efficient systems capable of solving real-world problems and responding in real-time. We propose the Syllable-Specific Temporal Encoding (SSTE) to learn vocal sequences in a reservoir of Izhikevich neurons, by forming associations between exclusive input activities and their corresponding syllables in the sequence. Our model converts the audio signals to cochleograms using the CAR-FAC model to simulate a brain-like auditory learning and memorization process. The reservoir is trained using a hardware-friendly approach to FORCE learning. Reservoir computing could yield associative memory dynamics with far less computational complexity compared to RNNs. The SSTE-based learning enables competent accuracy and stable recall of spatiotemporal sequences with fewer reservoir inputs compared with existing encodings in the literature for similar purpose, offering resource savings. The encoding points to syllable onsets and allows recalling from a desired point in the sequence, making it particularly suitable for recalling subsets of long vocal sequences. The SSTE demonstrates the capability of learning new signals without forgetting previously memorized sequences and displays robustness against occasional noise, a characteristic of real-world scenarios. The components of this model are configured to improve resource consumption and computational intensity, addressing some of the cost-efficiency issues that might arise in future implementations aiming for compactness and real-time, low-power operation. Overall, this model proposes a brain-inspired pattern generation network for vocal sequences that can be extended with other bio-inspired computations to explore their potentials for brain-like auditory perception. Future designs could inspire from this model to implement embedded devices that learn vocal sequences and recall them as needed in real-time. Such systems could acquire language and speech, operate as artificial assistants, and transcribe text to speech, in the presence of natural noise and corruption on audio data.

Symbol Acquisition from Motion Patterns and Their Sparce Coding by an Associative Memory Model with Self-organizing Nonmonotonicity

As has been claimed by Barlow [2], and reported by some recent neuro-physiological researches, at higher levels in the hierarchy of representations in the brain, sparse coding is adopted. Sparse coding is a kind of neural representation in which a very small number of neurons fire selectively. Because of the small overlaps between the firing patterns, the codes have the property of uniform metric, which may correspond to abstract symbolic representation of physical patterns [6]. We propose here a system that generates sparse codes of concepts of motions from accumulated feature vectors of observed motion patterns, by extending our previous research [11]. We propose an associative memory dynamics model with a self-organizing nonmonotonic activation function, which automatically finds out the hierarchical cluster structures in the stored motion patterns. Based on analysis of the dynamics of this model, we design an output function for the attractors, which can generate the sparse codes of the symbols of motion patterns.

Threshold control of chaotic neural network

The chaotic neural network constructed with chaotic neurons exhibits rich dynamic behaviour with a nonperiodic associative memory. In the chaotic neural network, however, it is difficult to distinguish the stored patterns in the output patterns because of the chaotic state of the network. In order to apply the nonperiodic associative memory into information search, pattern recognition etc. it is necessary to control chaos in the chaotic neural network. We have studied the chaotic neural network with threshold activated coupling, which provides a controlled network with associative memory dynamics. The network converges to one of its stored patterns or/and reverse patterns which has the smallest Hamming distance from the initial state of the network. The range of the threshold applied to control the neurons in the network depends on the noise level in the initial pattern and decreases with the increase of noise. The chaos control in the chaotic neural network by threshold activated coupling at varying time interval provides controlled output patterns with different temporal periods which depend upon the control parameters.

1P1-S-007 連想記憶ダイナミクスのアトラクタ分岐によるシンボル創発とそれを用いた記憶システム(知能の創発力学的アプローチ,生活を支援するロボメカ技術のメガインテグレーション)

1P1-S-007 連想記憶ダイナミクスのアトラクタ分岐によるシンボル創発とそれを用いた記憶システム(知能の創発力学的アプローチ,生活を支援するロボメカ技術のメガインテグレーション)

Dynamic synchronization and chaos in an associative neural network with multiple active memories.

Associative memory dynamics in neural networks are generally based on attractors. Retrieval based on fixed-point attractors works if only one memory pattern is retrieved at the time, but cannot enable the simultaneous retrieval of more than one pattern. Stable phase-locking of periodic oscillations or limit cycle attractors leads to incorrect feature bindings if the simultaneously retrieved patterns share some of their features. We investigate retrieval dynamics of multiple active patterns in a network of chaotic model neurons. Several memory patterns are kept simultaneously active and separated from each other by a dynamic itinerant synchronization between neurons. Neurons representing shared features alternate their synchronization between patterns, thus multiplexing their binding relationships. Our model includes a mechanism for self-organized readout or decoding of memory pattern coherence in terms of short-term potentiation and short-term depression of synaptic weights.

Application of two-parameter dynamical replica theory to retrieval dynamics of associative memory with non-monotonic neurons

The two-parameter dynamical replica theory (2-DRT) is applied to investigate retrieval properties of non-monotonic associative memory, a model which lacks thermodynamic potential functions. 2-DRT reproduces dynamical properties of the model quite well, including the capacity and basin of attraction. Superretrieval state is also discussed in the framework of 2-DRT. The local stability condition of the superretrieval state is given, which provides a better estimate of the region in which superretrieval is observed experimentally than the self-consistent signal-to-noise analysis (SCSNA) does.

A geometrical analysis of the dynamics of associative memory

This paper proposes a geometrical method of analyzing associative memory models. The geometrical method has the following two features: first, the state transitions of the associative memory model are subdivided into a linear transformation part based on the connection weight matrix and a nonlinear transformation part based on the sign function; second, a flow defined on the spherical surface is introduced. The flow consists of the dynamics on the spherical surface generated by the connection weight matrix and represents the dynamics of the model. The properties of the flow change suddenly when the memory exceeds a certain limit. This paper analyzes the relation of this change of flow and of the stability of the memorized vector itself. This paper shows that the geometrical method not only is effective in the analysis of the associative memory models, but also gives a good insight into ways of improving associative memory models and facilitates the understanding of the concept formation. © 1997 Scripta Technica, Inc. Syst Comp Jpn, 28(5): 68–81, 1997

Dynamics of associative memory with a general transfer function.

Dynamics of associative memory with a general transfer function.

CONTROL OF THE COMPLEXITY OF ASSOCIATIVE MEMORY DYNAMICS BY NEURONAL ADAPTATION

An abstract neural network model of the Hopfield type is extended to incorporate neuronal adaptation by defining the state of a neuron in terms of two variables, activity and excitability. The model is formulated to represent the regulation of the firing rate of action potentials in a biological system via the neuron cell membrane afterhyperpolarization by the effect of intracellular calcium ion concentration on the conductance of calcium sensitive potassium channels. It is shown that the complexity, and thus the exploratory degree, of associative memory dynamics are controlled by neuronal adaptability. At low adaptability, the dynamics have fixed point attractors corresponding to direct memory retrieval. In a subsequent region of adaptability values, a simple limit cycle persists with frequency increasing with adaptability. The range of frequencies agrees with that observed for theta rhythms of activity in the brain. A higher degree of freedom of the associative process corresponding to more complex dynamics, either limit cycles of varying complexity and period or chaotic behaviour, results at higher adaptability. In the brain, the neuronal adaptability is regulated by neuromodulators which suppress adaptation and increase absolute firing rates of action potentials. An associative process can be started at low concentration of neuromodulators as an exploratory search of state space during which firing rates are low. As the concentration of neuromodulators increases, the state space search becomes simpler cyclic and more restricted, and firing rates increase. Eventually, a particular stored state is retrieved and its activity is high. This correspondence between the complexity of associative memory dynamics and the concentration of neuromodulators is consistent with the observation for Alzheimer's disease of selective degeneracy of neurons releasing the neuromodulator acetylcholine. In an artificial neural network, inclusion of adaptation among neuronal properties allows control of the degree of freedom of associative processes and thus extends the range of possible applications.

Retrieval dynamics of associative memory of the Hopfield type

The authors have investigated the dynamics of retrieval processes of an associative memory of the Hopfield type (1982). For synchronous dynamics, they have generalized the theory of Amari and Maginu (1988), which enables them to treat the intermediate processes of memory retrieval in terms of a few simple macrovariables to the finite-temperature case. The resulting phase diagram in the equilibrium limit agrees qualitatively well with that from equilibrium statistical mechanics. They have carried out Monte Carlo simulations to clarify the limit of applicability of their theory. They have found that their basic assumption, an independent Gaussian distribution of the noise term with a time-dependent variance, is satisfied if the network succeeds in retrieval. When retrieval fails, the distribution of noise is non-Gaussian from very early stages of time development. For asynchronous dynamics, they propose a time-dependent Ginzburg-Landau approach, which simply expresses a downhill motion of the network in the free energy landscape. The resulting flow diagram in a phase space describes the behaviour of the network when it is close to equilibrium.

A neural network model of the dynamics of a short-term memory system in the temporal cortex

The temporal area TE of the monkey contains a group of neurons which seems to act as the short-term memory by a sustained firing, where the stored information seems to be retained by a certain kind of equilibrium of a dynamical system. The behavior of this group of neurons cannot be accounted for by the traditional dynamical system realized by the existing neural network models. This paper discusses the short-term memory system in the temporal area, as well as the dynamics of associative memory, and proposes the neural network model which realizes the same dynamical behavior as that of area TE. This model differs essentially from the traditional models in that the composing unit has non-monotonic input/output characteristics and a significant property in analyzing the behavior and the mechanics of the memory circuit in the brain.