Associative Memory Properties Research Articles

Active oscillatory associative memory

Traditionally, physical models of associative memory assume conditions of equilibrium. Here, we consider a prototypical oscillator model of associative memory and study how active noise sources that drive the system out of equilibrium, as well as nonlinearities in the interactions between the oscillators, affect the associative memory properties of the system. Our simulations show that pattern retrieval under active noise is more robust to the number of learned patterns and noise intensity than under passive noise. To understand this phenomenon, we analytically derive an effective energy correction due to the temporal correlations of active noise in the limit of short correlation decay time. We find that active noise deepens the energy wells corresponding to the patterns by strengthening the oscillator couplings, where the more nonlinear interactions are preferentially enhanced. Using replica theory, we demonstrate qualitative agreement between this effective picture and the retrieval simulations. Our work suggests that the nonlinearity in the oscillator couplings can improve memory under nonequilibrium conditions.

Thermodynamics of bidirectional associative memories

In this paper we investigate the equilibrium properties of bidirectional associative memories (BAMs). Introduced by Kosko in 1988 as a generalization of the Hopfield model to a bipartite structure, the simplest architecture is defined by two layers of neurons, with synaptic connections only between units of different layers: even without internal connections within each layer, information storage and retrieval are still possible through the reverberation of neural activities passing from one layer to another. We characterize the computational capabilities of a stochastic extension of this model in the thermodynamic limit, by applying rigorous techniques from statistical physics. A detailed picture of the phase diagram at the replica symmetric level is provided, both at finite temperature and in the noiseless regimes. Also for the latter, the critical load is further investigated up to one step of replica symmetry breaking. An analytical and numerical inspection of the transition curves (namely critical lines splitting the various modes of operation of the machine) is carried out as the control parameters—noise, load and asymmetry between the two layer sizes—are tuned. In particular, with a finite asymmetry between the two layers, it is shown how the BAM can store information more efficiently than the Hopfield model by requiring less parameters to encode a fixed number of patterns. Comparisons are made with numerical simulations of neural dynamics. Finally, a low-load analysis is carried out to explain the retrieval mechanism in the BAM by analogy with two interacting Hopfield models. A potential equivalence with two coupled Restricted Boltmzann Machines is also discussed.

A Complex-Valued Oscillatory Neural Network for Storage and Retrieval of Multidimensional Aperiodic Signals.

Recurrent neural networks with associative memory properties are typically based on fixed-point dynamics, which is fundamentally distinct from the oscillatory dynamics of the brain. There have been proposals for oscillatory associative memories, but here too, in the majority of cases, only binary patterns are stored as oscillatory states in the network. Oscillatory neural network models typically operate at a single/common frequency. At multiple frequencies, even a pair of oscillators with real coupling exhibits rich dynamics of Arnold tongues, not easily harnessed to achieve reliable memory storage and retrieval. Since real brain dynamics comprises of a wide range of spectral components, there is a need for oscillatory neural network models that operate at multiple frequencies. We propose an oscillatory neural network that can model multiple time series simultaneously by performing a Fourier-like decomposition of the signals. We show that these enhanced properties of a network of Hopf oscillators become possible by operating in the complex-variable domain. In this model, the single neural oscillator is modeled as a Hopf oscillator, with adaptive frequency and dynamics described over the complex domain. We propose a novel form of coupling, dubbed “power coupling,” between complex Hopf oscillators. With power coupling, expressed naturally only in the complex-variable domain, it is possible to achieve stable (normalized) phase relationships in a network of multifrequency oscillators. Network connections are trained either by Hebb-like learning or by delta rule, adapted to the complex domain. The network is capable of modeling N-channel electroencephalogram time series with high accuracy and shows the potential as an effective model of large-scale brain dynamics.

Place Cell Networks in Pre-weanling Rats Show Associative Memory Properties from the Onset of Exploratory Behavior.

Place cells are hippocampal pyramidal cells that are active when an animal visits a restricted area of the environment, and collectively their activity constitutes a neural representation of space. Place cell populations in the adult rat hippocampus display fundamental properties consistent with an associative memory network: the ability to 1) generate new and distinct spatial firing patterns when encountering novel spatial contexts or changes in sensory input (“remapping”) and 2) reinstate previously stored firing patterns when encountering a familiar context, including on the basis of an incomplete/degraded set of sensory cues (“pattern completion”). To date, it is unknown when these spatial memory responses emerge during brain development. Here, we show that, from the age of first exploration (postnatal day 16) onwards, place cell populations already exhibit these key features: they generate new representations upon exposure to a novel context and can reactivate familiar representations on the basis of an incomplete set of sensory cues. These results demonstrate that, as early as exploratory behaviors emerge, and despite the absence of an adult-like grid cell network, the developing hippocampus processes incoming sensory information as an associative memory network.

Information retrieval with unambiguous output

The main problem in information storage has previously been how large amounts of data can be stored. However, the technological development over the years has been able to give rather satisfactory answers to this problem. Recently, the focus has shifted towards determining how stored information can be efficiently retrieved. This problem is addressed in an article by E. Yaakobi and J. Bruck (2012), where information retrieval in associative memories is studied. In this paper, we focus on the case where the retrieved information unit is unambiguous. In particular, we present characterizations and study various extremal properties of such associative memories. Moreover, the algorithmic complexity of certain naturally rising problems is considered.

Anatomical constraints on lateral competition in columnar cortical architectures.

Competition is a well-studied and powerful mechanism for information processing in neuronal networks, providing noise rejection, signal restoration, decision making and associative memory properties, with relatively simple requirements for network architecture. Models based on competitive interactions have been used to describe the shaping of functional properties in visual cortex, as well as the development of functional maps in columnar cortex. These models require competition within a cortical area to occur on a wider spatial scale than cooperation, usually implemented by lateral inhibitory connections having a longer range than local excitatory connections. However, measurements of cortical anatomy reveal that the spatial extent of inhibition is in fact more restricted than that of excitation. Relatively few models reflect this, and it is unknown whether lateral competition can occur in cortical-like networks that have a realistic spatial relationship between excitation and inhibition. Here we analyze simple models for cortical columns and perform simulations of larger models to show how the spatial scales of excitation and inhibition can interact to produce competition through disynaptic inhibition. Our findings give strong support to the direct coupling effect-that the presence of competition across the cortical surface is predicted well by the anatomy of direct excitatory and inhibitory coupling and that multisynaptic network effects are negligible. This implies that for networks with short-range inhibition and longer-range excitation, the spatial extent of competition is even narrower than the range of inhibitory connections. Our results suggest the presence of network mechanisms that focus on intra-rather than intercolumn competition in neocortex, highlighting the need for both new models and direct experimental characterizations of lateral inhibition and competition in columnar cortex.

Sinusoidal modulation control method in a chaotic neural network

Chaotic neural networks (CNNs) have chaotic dynamic associative memory properties: The memory states appear non-periodically, and cannot be converged to a stored pattern. Thus, it is necessary to control chaos in a CNN in order to recognize associative memory. In this paper, a novel control method, the sinusoidal modulation control method, has been proposed to control chaos in a CNN. In this method, a sinusoidal wave simplified from brain waves is used as a control signal to modulate a parameter of the CNN. The simulation results demonstrate the effectiveness of this control method. The controlled CNN can be applied to information processing. Moreover, the method provides a way to associate brain waves by controlling CNNs.

Storage and retrieval of ultrametric patterns in a network of CA1 neurons of the hippocampus

We explore the possibility of storage and retrieval of ultrametrically organized patterns in hippocampus, the part of the brain devoted to the memory processes. The ultrametric structure has been chosen for having a good representation of the categories of memory. The storage and retrieval process is the one typical of the hippocampus and it is based on the dynamic of the CA1 neurons under the input from the neurons of the Enthorinal cortex and the Ca3 system. We explore if this real system of neurons exhibits the property of associative memory introduced since a long time in the artificial neural networks. We study how the performance is dependent on the deviation of the system of patterns from ultrametricity. The evolution of the system is simulated by means of a parallel computer and the statistics of storage and retrieval is investigated.

Olfactory Coding: Random Scents Make Sense

Stimulating arbitrary collections of as few as 300 neurons in the primary olfactory cortex of mice suffices for associative learning independent of any odor stimulation. Thus, programmed spatial relationships may not exist in piriform cortex, making flexible random associations the rule.

Lattice independent component analysis for functional magnetic resonance imaging

We introduce a lattice independent component analysis (LICA) unsupervised scheme to functional magnetic resonance imaging (fMRI) data analysis. LICA is a non-linear alternative to independent component analysis (ICA), such that ICA’s statistical independent sources correspond to LICA’s lattice independent sources. In this paper, LICA uses an incremental lattice source induction algorithm (ILSIA) to induce the lattice independent sources from the input dataset. The ILSIA computes a set of Strongly Lattice Independent vectors using properties of lattice associative memories regarding Lattice Independence and Chebyshev best approximation. The lattice independent sources constitute a set of Affine Independent vectors that define a simplex covering the input data. LICA carries out data linear unmixing based on the lattice independent sources basis. Therefore, LICA is a hybrid combination of a non-linear lattice based component and a linear unmixing component. The principal advantage over ICA is that LICA does not impose any probabilistic model assumptions on the data sources. We compare LICA with ICA in two case studies. Firstly, including simulated fMRI data, LICA discovers the spatial location of meaningful sources with less ambiguity than ICA. Secondly, including real data from an auditory stimulation experiment, LICA improves over some state of the art ICA variants discovering the activation patterns detected by Statistical Parametric Mapping (SPM) on the same data.

Attractor dynamics in a modular network model of the cerebral cortex

Computational models of cortical associative memory often take a top-down approach. We have previously described such an abstract model with a hypercolumnar structure. Here we explore a similar, biophysically detailed but subsampled network model of neocortex. We study how the neurodynamics and associative memory properties of this biophysical model relate to the abstract model as well as to experimental data. The resulting network exhibits attractor dynamics; pattern completion and pattern rivalry. It reproduces several features of experimentally observed local UP states, as well as oscillatory behavior on the gamma and theta time scales observed in the cerebral cortex.

Retrieval Property of Associative Memory Based on Inverse Function Delayed Neural Networks

Self-connection can enlarge the memory capacity of an associative memory based on the neural network. However, the basin size of the embedded memory state shrinks. The problem of basin size is related to undesirable stable states which are spurious. If we can destabilize these spurious states, we expect to improve the basin size. The inverse function delayed (ID) model, which includes the Bonhoeffer-van der Pol (BVP) model, has negative resistance in its dynamics. The negative resistance of the ID model can destabilize the equilibrium states on certain regions of the conventional neural network. Therefore, the associative memory based on the ID model, which has self-connection in order to enlarge the memory capacity, has the possibility to improve the basin size of the network. In this paper, we examine the fundamental characteristics of an associative memory based on the ID model by numerical simulation and show the improvement of performance compared with the conventional neural network.

Reconstruction of Patterns from Noisy Inputs Using Morphological Associative Memories

Morphological neural networks are based on a new paradigm for neural computing. Instead of adding the products of neural values and corresponding synaptic weights, the basic neural computation in a morphological neuron takes the maximum or minimum of the sums of neural values and their corresponding synaptic weights. By taking the maximum (or minimum) of sums instead of the sum of products, morphological neuron computation is nonlinear before thresholding. As a consequence, the properties of morphological neural networks are drastically different than those of traditional neural network models. In this paper we restrict our attention to morphological associative memories. After a brief review of morphological neural computing and a short discussion about the properties of morphological associative memories, we present new methodologies and associated theorems for retrieving complete stored patterns from noisy or incomplete patterns using morphological associative memories. These methodologies are derived from the notions of morphological independence, strong independence, minimal representations of patterns vectors, and kernels. Several examples are provided in order to illuminate these novel concepts.

Analysis of one-dimensional stochastic finite elements using neural networks

This paper explores the applicability of neural networks for analyzing the uncertainty spread of structural responses under the presence of one-dimensional random fields. Specifically, the neural network is intended to be a partial surrogate of the structural model needed in a Monte Carlo simulation, due to its associative memory properties. The network is trained with some pairs of input and output data obtained by some Monte Carlo simulations and then used in substitution of the finite element solver. In order to minimize the size of the networks, and hence the number of training pairs, the Karhunen–Loéve decomposition is applied as an optimal feature extraction tool. The Monte Carlo samples for training and validation are also generated using this decomposition. The Nyström technique is employed for the numerical solution of the Fredholm integral equation. The radial basis function (RBF) network was selected as the neural device for learning the input/output relationship due to its high accuracy and fast training speed. The analysis shows that this approach constitutes a promising method for stochastic finite element analysis inasmuch as the error with respect to the Monte Carlo simulation is negligible.

The declarative/procedural model of lexicon and grammar.

Our use of language depends upon two capacities: a mental lexicon of memorized words and a mental grammar of rules that underlie the sequential and hierarchical composition of lexical forms into predictably structured larger words, phrases, and sentences. The declarative/procedural model posits that the lexicon/grammar distinction in language is tied to the distinction between two well-studied brain memory systems. On this view, the memorization and use of at least simple words (those with noncompositional, that is, arbitrary form-meaning pairings) depends upon an associative memory of distributed representations that is subserved by temporal-lobe circuits previously implicated in the learning and use of fact and event knowledge. This "declarative memory" system appears to be specialized for learning arbitrarily related information (i.e., for associative binding). In contrast, the acquisition and use of grammatical rules that underlie symbol manipulation is subserved by frontal/basal-ganglia circuits previously implicated in the implicit (nonconscious) learning and expression of motor and cognitive "skills" and "habits" (e.g., from simple motor acts to skilled game playing). This "procedural" system may be specialized for computing sequences. This novel view of lexicon and grammar offers an alternative to the two main competing theoretical frameworks. It shares the perspective of traditional dual-mechanism theories in positing that the mental lexicon and a symbol-manipulating mental grammar are subserved by distinct computational components that may be linked to distinct brain structures. However, it diverges from these theories where they assume components dedicated to each of the two language capacities (that is, domain-specific) and in their common assumption that lexical memory is a rote list of items. Conversely, while it shares with single-mechanism theories the perspective that the two capacities are subserved by domain-independent computational mechanisms, it diverges from them where they link both capacities to a single associative memory system with broad anatomic distribution. The declarative/procedural model, but neither traditional dual- nor single-mechanism models, predicts double dissociations between lexicon and grammar, with associations among associative memory properties, memorized words and facts, and temporal-lobe structures, and among symbol-manipulation properties, grammatical rule products, motor skills, and frontal/basal-ganglia structures. In order to contrast lexicon and grammar while holding other factors constant, we have focused our investigations of the declarative/procedural model on morphologically complex word forms. Morphological transformations that are (largely) unproductive (e.g., in go-went, solemn-solemnity) are hypothesized to depend upon declarative memory. These have been contrasted with morphological transformations that are fully productive (e.g., in walk-walked, happy-happiness), whose computation is posited to be solely dependent upon grammatical rules subserved by the procedural system. Here evidence is presented from studies that use a range of psycholinguistic and neurolinguistic approaches with children and adults. It is argued that converging evidence from these studies supports the declarative/procedural model of lexicon and grammar.

Retrieval Properties of Bidirectional Associative Memories

We study a two-layer neural network made of N and M(N) neurons, producing a two-way association search for a family of p(N) patterns, where each pattern is a pair of two independent sub-categories of information having respectively N and M(N) components. In terms of the ratio γ=limN→∞M(N)/N, we study the retrieval capability of this network and show that there exists, at least, three regimes of association for which we determine the evolution of the threshold αc(γ) of the storage capacity α=limN→∞p(N)/N.

Color image correction for scanner and printer using B-spline CMAC neural networks.

The process of eliminating the color errors from the gamut mismatch, resolution conversion, and nonlinearity between scanner and printer is usually recognized as an essential issue of color reproduction. This paper presents a new formulation based on the generalized inverse plant control for the color error reduction process. In our formulation, the printer input and scanner output correspond to the input and output of a system plant, respectively. Obviously, if the printer input equals the scanner output, then there are no color errors involved in the entire system. In other words, the plant becomes an identity system. To achieve this goal, a plant generalized inverse should be identified and added to the original system. Since the system of a combination of both scanner and printer is highly nonlinear, CMAC-based neural networks, which have the capability to learn arbitrary nonlinearity, are applied to identify the plant generalized inverse. CMAC network is a perceptron-like feedforward structure with associative memory properties. Its memory requirements can be greatly reduced by the use of hash coding techniques. In order for CMAC networks to construct high-order, smooth, nonlinear plant inverse, more general CMAC addressing schemes have been proposed in conjunction with use of B-spline receptive functions. It is shown that B-spline CMAC networks learn orders of magnitude more rapidly than typical implementations of back propagation in the multilayered neural networks, due to the local nature of its weighting updating and the finite support of B-spline receptive field functions. Finally, a number of test samples are conducted to verify the effectiveness of the proposed method.

Modeling memory: what do we learn from attractor neural networks?

In this paper we summarize some of the main contributions of models of recurrent neural networks with associative memory properties. We compare the behavior of these attractor neural networks with empirical data from both physiology and psychology. This type of network could be used in models with more complex functions.

Orthogonal schemes for bidirectional associative memories

Two issues are addressed in this paper. Firstly, it investigates some important properties of bidirectional associative memories (BAM) and proposes an improved capacity estimate. Those properties are the encoding form of the input pattern pairs as sell as their decoding, the orthogonality of the pattern pairs, the similarity of associated patterns, and the density of the pattern pairs. Secondly, it proposes an implementation approach to improve the storage capacity. The approach embraces three proposed methods, i.e., the bipolar-orthogonal augmentation, the set partition, and the combined method. Along with those methods is the construction of the set of bipolar orthogonal patterns.

1-D MAPS, CHAOS AND NEURAL NETWORKS FOR INFORMATION PROCESSING

An application of complex dynamics and chaos in neural networks to information processing is studied. Mathematical models based on piecewise-linear maps implementing basic functions of information processing via complex dynamics and chaos are discussed. Realizations of these models by neural networks are presented. In contrast to other methods of using neural networks and associative memory to store information, the information is stored in dynamical attractors such as limit cycles, rather than equilibrium points. Retrieval of information corresponds to getting the state into the basin of attraction of the attractor. We show that noise-corrupted information or partial information are sufficient to drive the state into the basin of attraction of the attractor, thus these systems exhibit the property of associative memory.