Fundamental Memories Research Articles

Associative morphological memories based on variations of the kernel and dual kernel methods

Morphological associative memories (MAMs) belong to the class of morphological neural networks. The recording scheme used in the original MAM models is similar to the correlation recording recipe. Recording is achieved by means of a maximum ( M XY model) or minimum ( W XY model) of outer products. Notable features of autoassociative morphological memories (AMMs) include optimal absolute storage capacity and one-step convergence. Heteroassociative morphological memories (HMMs) do not have these properties and are not very well understood. The fixed points of AMMs can be characterized exactly in terms of the original patterns. Unfortunately, AMM fixed points include a large number of spurious memories. In this paper, we combine the M XX model and variations of the kernel method to produce new autoassociative and heteroassociative memories. We also introduce a dual kernel method. A new, dual model is given by a combination of the W XX model and a variation of the dual kernel method. The new MAM models exhibit better error correction capabilities than M XX and W XX and a reduced number of spurious memories which can be easily described in terms of the fundamental memories.

Typical error pattern recovery of the Hopfield memory under error-tolerant conditions

We lower-bound the error-correcting capabilities of the Hopfield (1982) network under error-tolerant conditions, given some distorted version of a fundamental memory with a random (typical) error pattern. Our main result is that for any sufficiently small /spl alpha/, and for any /spl rho/<1/2, a Hopfield memory with n neurons can store /spl alpha/n fundamental memories, such that the following is satisfied in probability: a distorted fundamental memory with /spl rho/n errors, located at random positions may be corrected up to a residual error rate of exp{-1/2/spl alpha/}.

Long-term attraction in higher order neural networks

Recent results on the memory storage capacity of higher order neural networks indicate a significant improvement compared to the limited capacity of the Hopfield model. However, such results have so far been obtained under the restriction that only a single iteration is allowed to converge. This paper presents a indirect convergence (long-term attraction) analysis of higher order neural networks. Our main result is that for any kappa(d)<d!2(d-1)/(2d)!, and 0< or =rho<1/2, a Hebbian higher order neural network of order d with n neurons can store a random set of kappa(d)n(d)/log n fundamental memories such that almost all memories have an attraction radius of size rhon. If kappa(d)<d!2(d-1)/((2d)!(d+1)), then all memories possess this property simultaneously. It indicates that the lower bounds on the long-term attraction capacities are larger than the corresponding direct convergence capacities by a factor of 1/(1-2rho) (2d). In addition we upper bound the convergence rate (number of iterations required to converge). This bound is asymptotically independent of n. Similar results are obtained for zero diagonal higher order neural networks.

Recurrent neural nets as dynamical Boolean systems with application to associative memory

Discrete-time/discrete-state recurrent neural networks are analyzed from a dynamical Boolean systems point of view in order to devise new analytic and design methods for the class of both single and multilayer recurrent artificial neural networks. With the proposed dynamical Boolean systems analysis, we are able to formulate necessary and sufficient conditions for network stability which are more general than the well-known but restrictive conditions for the class of single layer networks: (1) symmetric weight matrix with (2) positive diagonal and (3) asynchronous update. In terms of design, we use a dynamical Boolean systems analysis to construct a high performance associative memory. With this Boolean memory, we can guarantee that all fundamental memories are stored, and also guarantee the size of the basin of attraction for each fundamental memory.

A multimodule neural network model and the estimate of its nature as associative memory

AbstractA modular neural network model, “Cross‐Coupled Hopfield Nets with Many‐to‐Many Mapping Internetworks (CCHN‐MMMI),” in which the total consumed energy is defined as a linear sum of energy functions for subnetworks and their interactions, is proposed herein. This model can be applied to cases in which many‐to‐many mappings are necessary to mutually relate states of modules. The network dynamics are derived first, and then through computer simulations, the characteristics of the 2‐module CCHN‐MMMI are studied as associative memory. In one simulation, the effects of the introduction of explicitly modular architecture and a comparison of the results to those produced by conventional autocorrelation associative memory models.Another simulation shows that the model can store and recall pairs of character patterns successfully even if many‐to‐many mapping relations are necessary to relate the pairs to each other. A quantitative investigation as to how the recall dynamics deteriorate with an increasing number of randomly chosen fundamental memories is made in order to access the model as associative. The results indicate that internetworks between modules function effectively to prevent the entire network from recalling spurious memories. When multilayered networks with nonlinear hidden units are employed as internetworks in the model, the association performance is even more greatly improved as compared with conventional autocorrelation associative memory model.

Nondirect convergence radius and number of iterations of the Hopfield associative memory

Considers a Hopfield associative memory consisting of n neurons, designed to store an m-set of n-dimensional /spl plusmn/1 statistically independent uniformly distributed random vectors (fundamental memories), using a connection matrix, constructed by the usual Hebbian rule. Previous results have indicated that the maximal value of m, such that almost all m vectors are stable points of the memory, in probability (i.e., with probability approaching one as n approaches infinity), is n/(2 log n)(n/(4 log n) if all m vectors must be stable simultaneously, in probability). Previous work further analyzed the direct convergence (i.e., convergence in one iteration) error-correcting power of the Hopfield memory. The present authors rigorously analyze the general case of nondirect convergence, and prove that in the m=n/(2 log n) case, independently of the operation mode used (synchronous or asynchronous), almost all memories have an attraction radius of size n/2 around them (in the n/(4 log n) case, all memories have such an attraction radius, in probability). This result, which was conjectured in the past but was never proved rigorously, combined with an old converse result that the network cannot store more than n/(2 log n)(n/(4 log n)) fundamental memories, gives a full picture of the error-correcting power of the Hebbian Hopfield network. The authors also upper bound the number of iterations required to achieve convergence.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Characteristics of the Hopfield associative memory utilizing isomorphism relations

Isomorphism relations are utilized to analyze the Hopfield associative memory. When the number of fundamental memories m=/<3, it is proved that two Hopfield associative memories are isomorphic if they have the same mutual distances between the fundamental memories. The number of stable states and the synchronous convergence time of a Hopfield associative memory are shown to be less than or equal to 2 to the power 2(m-1) and 4 to the power 2(m-1), respectively, where m>/=1.

Convergence results in an associative memory model

This paper presents rigorous mathematical proofs for some observed convergence phenomena in an associative memory model introduced by Hopfield (based on Hebbian rules) for storing a number of random n-bit patterns. The capability of the model to correct a linear number of random errors in a bit pattern has been established earlier, but the existence of a large domain of attraction (correcting a linear number of arbitrary errors) has not been proved. We present proofs for the following: • • When m , the number of patterns stored, is less than n/(4 log n) , the fundamental memories have a domain of attraction of radius ρn with ρ = 0.024, and the algorithm converges in time O (log log n) . • • When m = α n (with α small), all patterns within a distance ρn from a fundamental memory end up, in constant time, within a distance ϵn from the fundamental memory, where ϵ is about e −1/4α We also extend somewhat Newman's description of the “energy landscape,” and prove the existence of an exponential number of stable states (extraneous memories) with convergence properties similar to those of the fundamental memories.

The capacity of the Hopfield associative memory

Techniques from coding theory are applied to study rigorously the capacity of the Hopfield associative memory. Such a memory stores n -tuple of \pm 1 's. The components change depending on a hard-limited version of linear functions of all other components. With symmetric connections between components, a stable state is ultimately reached. By building up the connection matrix as a sum-of-outer products of m fundamental memories, one hopes to be able to recover a certain one of the m memories by using an initial n -tuple probe vector less than a Hamming distance n/2 away from the fundamental memory. If m fundamental memories are chosen at random, the maximum asympotic value of m in order that most of the m original memories are exactly recoverable is n/(2 \log n) . With the added restriction that every one of the m fundamental memories be recoverable exactly, m can be no more than n/(4 \log n) asymptotically as n approaches infinity. Extensions are also considered, in particular to capacity under quantization of the outer-product connection matrix. This quantized memory capacity problem is closely related to the capacity of the quantized Gaussian channel.