Neural Associative Memory Research Articles

A modular approach to language production: Models and facts

Numerous cortical disorders affect language. We explore the connection between the observed language behavior and the underlying substrates by adopting a neurocomputational approach. To represent the observed trajectories of the discourse in patients with disorganized speech and in healthy participants, we design a graphical representation for the discourse as a trajectory that allows us to visualize and measure the degree of order in the discourse as a function of the disorder of the trajectories. Our work assumes that many of the properties of language production and comprehension can be understood in terms of the dynamics of modular networks of neural associative memories. Based upon this assumption, we connect three theoretical and empirical domains: (1) neural models of language processing and production, (2) statistical methods used in the construction of functional brain images, and (3) corpus linguistic tools, such as Latent Semantic Analysis (henceforth LSA), that are used to discover the topic organization of language. We show how the neurocomputational models intertwine with LSA and the mathematical basis of functional neuroimaging. Within this framework we describe the properties of a context-dependent neural model, based on matrix associative memories, that performs goal-oriented linguistic behavior. We link these matrix associative memory models with the mathematics that underlie functional neuroimaging techniques and present the “functional brain images” emerging from the model. This provides us with a completely “transparent box” with which to analyze the implication of some statistical images. Finally, we use these models to explore the possibility that functional synaptic disconnection can lead to an increase in connectivity between the representations of concepts that could explain some of the alterations in discourse displayed by patients with schizophrenia.

Neural associative memories and sparse coding

The theoretical, practical and technical development of neural associative memories during the last 40 years is described. The importance of sparse coding of associative memory patterns is pointed out. The use of associative memory networks for large scale brain modeling is also mentioned.

Feedfoward and feedback adaptive controls for Continuously Variable Transmissions

Continuously variable transmissions (CVT) require a feedback controller to reach a targeted transmission speed ratio by commanding pressures on moveable sheaves. The implementation of two on-line identification algorithms are presented to improve the performances and the robutness of the CVT. To compensate for the slow drift of the electro-hydraulic valves during the warm-up period of the transmission, an adaptive feedforward strategy is implemented with the Dasgupta-Huang Outer Bounding Ellipsoid (DHOBE); because of highly correlated input data, a regularization procedure is added, giving the rDHOBE. To compensate for the effect of wear on the pulleys, an improved feedback linearization design is proposed with an on-line adaption of the main nonlinear map in a neural associative memory. The adaptation is required because the stability condition of the feedback controller depends of the accuracy of one of the estimated non-linear maps. The experiments show clear advantages for such adaptive controls over non-adaptive ones.

Adaptive control of a continuously variable transmission subject to wear

Continuously variable transmissions (CVT) require an electronic controller for real application: it must reach a targeted transmission speed ratio in spite of wear. Based on our recent model based for high speed steel belt CVT, an improved feedback linearization design is proposed with an on-line adaption of the main nonlinear map in a neural associative memory. The adaptation is required because the stability condition depends of the accuracy of the estimated nonlinear map. Experimental results show that the adaptation allows the speed ratio controller to remain stable regardless of the wear accumulated by the CVT transmission.

Personalized Search Strategies for Spatial Information on the Web

A spatial information search strategy based on the Bidirectional Neural Associative Memory model produces search results that are sensitive to user preferences.

Neural Associative Memory with Optimal Bayesian Learning

Neural associative memories are perceptron-like single-layer networks with fast synaptic learning typically storing discrete associations between pairs of neural activity patterns. Previous work optimized the memory capacity for various models of synaptic learning: linear Hopfield-type rules, the Willshaw model employing binary synapses, or the BCPNN rule of Lansner and Ekeberg, for example. Here I show that all of these previous models are limit cases of a general optimal model where synaptic learning is determined by probabilistic Bayesian considerations. Asymptotically, for large networks and very sparse neuron activity, the Bayesian model becomes identical to an inhibitory implementation of the Willshaw and BCPNN-type models. For less sparse patterns, the Bayesian model becomes identical to Hopfield-type networks employing the covariance rule. For intermediate sparseness or finite networks, the optimal Bayesian learning rule differs from the previous models and can significantly improve memory performance. I also provide a unified analytical framework to determine memory capacity at a given output noise level that links approaches based on mutual information, Hamming distance, and signal-to-noise ratio.

Learning Associative Memories by Error Backpropagation

In this paper, a method for the design of Hopfield networks, bidirectional and multidirectional associative memories with asymmetric connections, is proposed. The given patterns can be assigned as locally asymptotically stable equilibria of the network by training a single-layer feedforward network. It is shown that the robustness in respect to acceptable noise in the input of the constructed networks is enhanced as the memory dimension increases and weakened as the number of the stored patterns grows. More important is that the remembered patterns are not necessarily of binary forms. Neural associative memories for storing gray-level images are constructed based on the proposed method. Numerical simulations show that the proposed method is efficient for the design of Hopfield-type recurrent neural networks.

Color image associative memory on a class of Cohen–Grossberg networks

In this paper, neural associative memories for storing gray-scale and true color images are presented based on a class of reduced Cohen–Grossberg neural networks. Some fundamental conditions for endowing the networks with retrieval properties are proposed. Moreover, a system designing procedure is developed by using matrix decomposition. Numerical simulations show that the constructed networks can act as reliable noise-reducing systems for storing and retrieving color images.

Artificial neural networks in hardware: A survey of two decades of progress

This article presents a comprehensive overview of the hardware realizations of artificial neural network (ANN) models, known as hardware neural networks (HNN), appearing in academic studies as prototypes as well as in commercial use. HNN research has witnessed a steady progress for more than last two decades, though commercial adoption of the technology has been relatively slower. We study the overall progress in the field across all major ANN models, hardware design approaches, and applications. We outline underlying design approaches for mapping an ANN model onto a compact, reliable, and energy efficient hardware entailing computation and communication and survey a wide range of illustrative examples. Chip design approaches (digital, analog, hybrid, and FPGA based) at neuronal level and as neurochips realizing complete ANN models are studied. We specifically discuss, in detail, neuromorphic designs including spiking neural network hardware, cellular neural network implementations, reconfigurable FPGA based implementations, in particular, for stochastic ANN models, and optical implementations. Parallel digital implementations employing bit-slice, systolic, and SIMD architectures, implementations for associative neural memories, and RAM based implementations are also outlined. We trace the recent trends and explore potential future research directions.

A Study on Associative Neural Memories

Memory plays a major role in Artificial Neural Networks. Without memory, Neural Network can not be learned itself. One of the primary concepts of memory in neural networks is Associative neural memories. A survey has been made on associative neural memories such as Simple associative memories (SAM), Dynamic associative memories (DAM), Bidirectional Associative memories (BAM), Hopfield memories, Context Sensitive Auto-associative memories (CSAM) and so on. These memories can be applied in various fields to get the effective outcomes. We present a study on these associative memories in artificial neural networks.

Optimal and robust design of brain-state-in-a-box neural associative memories

This paper presents a new optimization approach to the design of associative memories via the brain-state-in-a-box (BSB) neural network. The optimization approach proposed in this paper provides the large and uniform domains of attraction of the prototype patterns, the large robustness margin for the weight matrix of the perturbed BSB neural network, the asymptotic stability of the prototype patterns, and the global stability of the BSB neural network. Based on some known qualitative properties of the BSB neural network and theoretical results presented in this paper, a synthesis method of the BSB-based associative memory is established. The synthesis method presented in this paper is given in the form of a linear matrix inequality-based optimization problem, which can be efficiently solved by a readily available software. Design examples are given to demonstrate the applicability of the proposed method and to compare with the existing synthesis methods.

Reduced order multiport parallel and multidirectional neural associative memories

This paper proposes multiport parallel and multidirectional intraconnected associative memories of outer product type with reduced interconnections. Some new reduced order memory architectures such as k-directional and k-port parallel memories are suggested. These architectures are, also, very suitable for implementation of spatio-temporal sequences and multiassociative memories. It is shown that in the proposed memory architectures, a substational reduction in interconnections is achieved if the actual length of original N-bit long vectors is subdivided into k sublengths. Using these sublengths, submemory matrices, T ( s ) or W ( s ), are computed, which are then intraconnected to form k-port parallel or k-directional memories. The subdivisions of N-bit long vectors into k sublengths save ((k-1) x 100) / k % of interconnections. It is shown, by means of an example, that more than 80% reduction in interconnections is achieved. Minimum limit in bits on k as well as maximum limit on subdivisions in k is determined. The topologies of reduced interconnectivity developed in this paper are symmetric in structure and can be used to scale up to larger systems. The underlying principal of construction, storage and retrieval processes of such associative memories has been analyzed. The effect of complexity of different levels of reduced interconnectivity on the quality of retrieval, signal to noise ratio, and storage capacity has been investigated. The model possesses analogies to biological neural structures and digital parallel port memories commonly used in parallel and multiprocessing systems.

Neural associative memories for the integration of language, vision and action in an autonomous agent

Language understanding is a long-standing problem in computer science. However, the human brain is capable of processing complex languages with seemingly no difficulties. This paper shows a model for language understanding using biologically plausible neural networks composed of associative memories. The model is able to deal with ambiguities on the single word and grammatical level. The language system is embedded into a robot in order to demonstrate the correct semantical understanding of the input sentences by letting the robot perform corresponding actions. For that purpose, a simple neural action planning system has been combined with neural networks for visual object recognition and visual attention control mechanisms.

Content-Based Framework For Component Libraries

The construction of large software systems from existing components requires efficient access to possibly large collections of such components. In this paper, we present a content-based framework to organize this large collection of components in a semi-automatic fashion, according to an extensible user-defined ontology. Neural associative memories are used for fast, similarity-based access. Relevant characteristics of components are extracted and stored as a signature in such an associative memory, and during retrieval the closest matches to the query are indicated in very short time. In addition to content-based characteristics such as keywords, function and variable names, comments, and the location of the component in the class hierarchy, this method can be easily expanded to include usage characteristics, resource requirements, or other task- or customer-specific criteria.

Memory Convergence and Optimization with Fuzzy PSO and ACS

Associative neural memories are models of biological phenomena that allow for the storage of pattern associations and the retrieval of the desired output pattern upon presentation of a possibly noisy or incomplete version of an input pattern. In this study, we introduce fuzzy swarm particle optimization technique for convergence of associative neural memories based on fuzzy set theory. A Fuzzy Particle Swarm Optimization (FPSO) consists of clustering of swarm's particle by applying fuzzy c-mean algorithm to attain the neighborhood best. We present a singular value decomposition method for the selection of efficient rule from a given rule base required to attain the global best. Finally, we illustrate the proposed method by virtue of some examples. Further, ant colony system ACS algorithm is used to study the Symmetric Traveling Salesman Problem TSP. The optimum parameters for this algorithm have to found by trial and error. The ACS parameters working in a designed subset of TSP instances has also been optimized by virtue of Particle Swarm Optimization PSO.

Neural Associative Memory and the Willshaw–Palm Probability Distribution

Previous asymptotic analyses of binary neural associative networks of Willshaw or Steinbuch type relied on a binomial approximation of the neurons' dendritic potentials. This approximation has been proven to be good only if the stored patterns are extremely sparse, for example, when the mean number of active units k per pattern vector scales with the logarithm of the vector size n. Recent promising results concerning storage capacity and retrieval efficiency for larger pattern activities $k > \log n$ have been doubted because here the binomial approximation can lead to a massive overestimation of performance. In this work I compute and characterize the exact Willshaw–Palm distribution of the dendritic potentials for hetero-association, auto-association, and fixed and random pattern activity. Comparing the raw and central moments of the Willshaw–Palm distribution to the moments of the corresponding binomial probability reveals that, asymptotically, the binomial approximation becomes exact for almost any sublinear pattern activity, including $k = O(n/\log^2 n)$. This verifies, for large networks, the existence of a wide high-performance parameter range as predicted by the approximative theory.

Closed-form Expressions for the Moments of the Binomial Probability Distribution

This work develops closed-form expressions for the raw and central moments of the binomial probability distribution. For this I first derive a recursive formula for the raw moments from the moment generating function. Then it is shown that the recursion involved is essentially the same as for the Stirling numbers of the second kind. From this fact it is then possible to derive the closed formulae. Finally, I discuss an application of these formulae to the analysis of neural associative memory.

Announcement/Call for Papers: 2nd International Conference on Neural Networks and Associative Memories (NNAM 2007)

Announcement/Call for Papers: 2nd International Conference on Neural Networks and Associative Memories (NNAM 2007)

Implicative Fuzzy Associative Memories

Associative neural memories are models of biological phenomena that allow for the storage of pattern associations and the retrieval of the desired output pattern upon presentation of a possibly noisy or incomplete version of an input pattern. In this paper, we introduce implicative fuzzy associative memories (IFAMs), a class of associative neural memories based on fuzzy set theory. An IFAM consists of a network of completely interconnected Pedrycz logic neurons with threshold whose connection weights are determined by the minimum of implications of presynaptic and postsynaptic activations. We present a series of results for autoassociative models including one pass convergence, unlimited storage capacity and tolerance with respect to eroded patterns. Finally, we present some results on fixed points and discuss the relationship between implicative fuzzy associative memories and morphological associative memories

Quantum learning for neural associative memories

Quantum information processing in neural structures results in an exponential increase of patterns storage capacity and can explain the extensive memorization and inferencing capabilities of humans. An example can be found in neural associative memories if the synaptic weights are taken to be fuzzy variables. In that case, the weights’ update is carried out with the use of a fuzzy learning algorithm which satisfies basic postulates of quantum mechanics. The resulting weight matrix can be decomposed into a superposition of associative memories. Thus, the fundamental memory patterns (attractors) can be mapped into different vector spaces which are related to each other via unitary rotations. Quantum learning increases the storage capacity of associative memories by a factor of 2 N , where N is the number of neurons.