Hopfield Model Of Associative Memory Research Articles

An environmental approach to developing and applying smart complexes of academic disciplines in professional training of future specialists

The paper discloses modern approaches to creating integrated information environments of SMART complexes of academic disciplines through integrating creative, authorial, non-verbal, encyclopaedic, information-and-communication, self-realization, self-assessment components in professional (vocational) and pre-university professional education. It lists the advantages of such complexes compared to e-textbooks and reveals the requirements for developing them. It highlights the ways of considering future specialists’ psycho-physiological development when selecting and structuring educational information. It recommends applying constructive equalization of students’ cognitive activity based on the Kosko’s quasi-neural network model in their designing an educational trajectory. It shows conditions for ensuring equal opportunities for students’ learning within such complexes. Both selection and structurization of such complexes’ educational information follow students’ psychological development in perceiving it and focus on critical feature patterns of the adaptive resonance theory and the Hopfield model for associative memory. The paper suggests evaluating students’ activities within such complexes by comparing each participant’s achievements with the parameters of completed projects, using the index method based on qualimetric measurements. It specifies the features of an environmental approach in developing such complexes; elaborating their educational material; determining types of learning tasks; creating means of monitoring students’ knowledge. It justifies the results of experimental work, which involved surveying 442 teachers and analyzes the influence of such complexes on the effectiveness of the educational process. It highlights the importance of introducing heuristic forms, methods and techniques of students’ learning to help students obtain, systematize and consolidate educational information and acquire practical skills in performing creative projects and professional tasks.

Non-convex Multi-species Hopfield Models

In this work we introduce a multi-species generalization of the Hopfield model for associative memory, where neurons are divided into groups and both inter-groups and intra-groups pair-wise interactions are considered, with different intensities. Thus, this system contains two of the main ingredients of modern Deep neural network architectures: Hebbian interactions to store patterns of information and multiple layers coding different levels of correlations. The model is completely solvable in the low-load regime with a suitable generalization of the Hamilton-Jacobi technique, despite the Hamiltonian can be a non-definite quadratic form of the magnetizations. The family of multi-species Hopfield model includes, as special cases, the 3-layers Restricted Boltzmann Machine (RBM) with Gaussian hidden layer and the Bidirectional Associative Memory (BAM) model.

High storage capacity in the Hopfield model with auto-interactions—stability analysis

Recent studies point to the potential storage of a large number of patterns in the celebrated Hopfield associative memory model, well beyond the limits obtained previously. We investigate the properties of new fixed points to discover that they exhibit instabilities for small perturbations and are therefore of limited value as associative memories. Moreover, a large deviations approach also shows that errors introduced to the original patterns induce additional errors and increased corruption with respect to the stored patterns.

On a Model of Associative Memory with Huge Storage Capacity

In [7] Krotov and Hopfield suggest a generalized version of the well-known Hopfield model of associative memory. In their version they consider a polynomial interaction function and claim that this increases the storage capacity of the model. We prove this claim and take the "limit" as the degree of the polynomial becomes infinite, i.e. an exponential interaction function. With this interaction we prove that model has an exponential storage capacity in the number of neurons, yet the basins of attraction are almost as large as in the standard Hopfield model.

Chaotic pseudo-orthogonalized Hopfield associative memory

A Hopfield associative memory (HAM) is a major model of associative memory using neural networks. The classical HAM model has a low storage capacity. A pseudo-orthogonalized HAM (POHAM) was therefore proposed to improve storage capacity by encoding the training patterns to be pseudo-orthogonal. In the present work, we examine a different property of POHAM. A chaotic HAM (CHAM) can explore embedded patterns, including training patterns. Although it is not desirable to recall patterns other than training patterns, at a minimum, the reversed patterns are recalled. A POHAM regards the training and reversed patterns as equivalent. To take advantage of this property, we propose a chaotic POHAM (CPOHAM). We evaluated a CHAM and a CPOHAM using computer simulations. The CPOHAM never recalled the reversed patterns. In addition, the CPOHAM recalled very few other pseudo-memories, such as mixture patterns, due to the orthogonality of the encoded training patterns, while the CHAM frequently recalled the mixture patterns.

Información, estabilidad y complejidad de aprendizaje en memorias asociativas

espanolEn este trabajo se estudia la relacion entre la canti-dad de informacion y la complejidad de aprendizaje en el modelo discreto de Hopfield de memoria asociativa. Mas especificamente, se analiza, desde un punto de vista estadistico, la relacion entre la probabilidad de que un estado del sistema sea un equilibrio estable (memoria) y un valor de entropia o incertidumbre asociado a el. Se realizan experimentos computacionales para corroborar estos resultados. EnglishThe relationship between amount of information and learning complexity is studied for the discrete Hopfield model of associative memory. More precisely, we analize, from a statistical point of view, the relation between the probability of a state of the system to be a stable equilib-rium (i.e. a memory) and a value of entropy or uncertainty associated to it. Computer experiments are made to confirm these results.

Statistical–Mechanical Analysis of Attractor Dynamics in a Hysteretic Neuron Network

The attractor dynamics of a neural network of neurons with a hysteretic response property are investigated by a statistical–mechanical approach and numerical simulations using the Hopfield associative-memory model. Exact macroscopic flow equations for the case that the number of stored patterns is finite are derived, and the stability of attractors is evaluated. It is shown that the hysteretic property improves the robustness of both memory patterns and mixed states against temperature but does not affect the qualitative structure of the phase diagram. In addition, it is numerically showed that as the number of stored patterns increases and the state of the system becomes frustrated, the relaxation time becomes very long. In contrast to the results of the previous studies, however, memory capacity was not improved by the hysteretic property.

Parallel and orthogonal stimulus in ultradiluted neural networks

Extending a model due to Derrida, Gardner, and Zippelius, we have studied the recognition ability of an extreme and asymmetrically diluted version of the Hopfield model for associative memory by including the effect of a stimulus in the dynamics of the system. We obtain exact results for the dynamic evolution of the average network superposition. The stimulus field was considered as proportional to the overlapping of the state of the system with a particular stimulated pattern. Two situations were analyzed, namely, the external stimulus acting on the initialization pattern (parallel stimulus) and the external stimulus acting on a pattern orthogonal to the initialization one (orthogonal stimulus). In both cases, we obtained the complete phase diagram in the parameter space composed of the stimulus field, thermal noise, and network capacity. Our results show that the system improves its recognition ability for parallel stimulus. For orthogonal stimulus two recognition phases emerge with the system locking at the initialization or stimulated pattern. We confront our analytical results with numerical simulations for the noiseless case T = 0.

Gauged neural network: Phase structure, learning, and associative memory

A gauge model of neural network is introduced, which resembles the Z(2) Higgs lattice gauge theory of high-energy physics. It contains a neuron variable S x = ± 1 on each site x of a 3D lattice and a synaptic-connection variable J x μ = ± 1 on each link ( x , x + μ ^ ) ( μ = 1 , 2 , 3 ) . The model is regarded as a generalization of the Hopfield model of associative memory to a model of learning by converting the synaptic weight between x and x + μ ^ to a dynamical Z(2) gauge variable J x μ . The local Z(2) gauge symmetry is inherited from the Hopfield model and assures us the locality of time evolutions of S x and J x μ and a generalized Hebbian learning rule. At finite “temperatures”, numerical simulations show that the model exhibits the Higgs, confinement, and Coulomb phases. We simulate dynamical processes of learning a pattern of S x and recalling it, and classify the parameter space according to the performance. At some parameter regions, stable column-layer structures in signal propagations are spontaneously generated. Mutual interactions between S x and J x μ induce partial memory loss as expected.

Categorization ability in a biologically motivated neural network

In this work, we study the categorization properties of an extreme and asymmetrically diluted version of the Hopfield model for associative memory when the effect of the refractory period of the neurons is taken into account in the dynamics of the system. The simplest way of modeling these refractory periods is by means of a time-dependent threshold that acts only on those neurons that emit a signal and favors them to be at rest during a given time interval. The dynamic equations are derived in the limit of extreme dilution, using an approach, that explicitly preserves the dependence of the system on its whole history. The categorization error is analyzed for different values of the parameters. In particular, we confront our analytical results with numerical simulations for the noiseless case T=0. When the number of examples or their correlations increases, the system always categorizes independently of the amplitude of the potential that mimics the effect of the refractory period.

Out-of-equilibrium dynamics of the hopfield model in its spin-glass phase

In this paper, we study numerically the out-of-equilibrium dynamics of the Hopfield model for associative memory inside its spin-glass phase. Aside from its interest as a neural network model, it can also be considered as a prototype of a fully connected magnetic system with randomness and frustration. By adjusting the ratio between the number of stored configurations p and the total number of neurons N, one can control the phase-space structure, whose complexity can vary between the simple mean-field ferromagnet (when p=1) and that of the Sherrington Kirkpatrick spin-glass model (for a properly taken limit of an infinite number of patterns). In particular, little attention has been devoted to the spin-glass phase of this model. In this paper, we analyze the two-time autocorrelation function, the decay of the magnetization and the distribution of overlaps between states. The results show that within the spin-glass phase of the model, the dynamics exhibits aging phenomena and presents features that suggest a non trivial breaking of replica symmetry.

Recognition ability in a biologically motivated neural network

In this work we study the storage capacity of an extreme and asymmetrically diluted version of the Hopfield model for associative memory when the effect of the refractory periods of the neurons is included through a time dependent threshold. We develop a new dynamical approach that allows us to take into account, in an approximative way, the dependence of the system on its whole history. We also compare our analytical results with those obtained through a very extensive numerical simulation.

Lower bounds on the restitution error in the Hopfield Model

This paper contains a first rigorous results at the direction of the converse to Newman's bounds [11] on the storage capacity for the Hopfield model of associative memory. Let $m$ denote the number of stored patterns and $N$ that of neurons. Assuming that $m/N\to \alpha$ as $N\to\infty $ we show that for $\alpha>0$ the memory necessarily commits a positive fraction of errors on memorized patterns that becomes superior to some threshold value (approximately 0.05) as $\alpha\to\infty $ .

INFLUENCE OF REFRACTORY PERIODS IN THE HOPFIELD MODEL

We study both analytically and numerically the effects of including refractory periods in the Hopfield model for associative memory. These periods are introduced in the dynamics of the network as thresholds that depend on the state of the neuron at the previous time. Both the retrieval properties and the dynamical behavior are analyzed, and we found that, depending on the value of the thresholds and on the the ratio α between the number of stored memories (p) and the total number of neurons (N), the system presents not only fixed points but also chaotic or cyclic orbits. The recognizing capability of the network in the cyclic region is also studied.

Fixed points in a Hopfield model with random asymmetric interactions.

We calculate analytically the average number of fixed points in the Hopfield model of associative memory when a random antisymmetric part is added to the otherwise symmetric synaptic matrix. Addition of the antisymmetric part causes an exponential decrease in the total number of fixed points. If the relative strength of the antisymmetric component is small, then its presence does not cause any substantial degradation of the quality of retrieval when the memory loading level is low. We also present results of numerical simulations which provide qualitative (as well as quantitative for some aspects) confirmation of the predictions of the analytic study. Our numerical results suggest that the analytic calculation of the average number of fixed points yields the correct value for the typical number of fixed points.

Retrieval characteristics of multiple site states networks with self-interaction

The Hopfield model of associative memory is modified by the inclusion of self-interaction and also by taking multiple site states networks. Computer simulations are presented to demonstrate the improvements in the retrieval characteristics as a result of the inclusion of self-interaction terms in an N = 401 network. They show that self-interaction enhances the storage capacity of a network, decreases the average error in retrieved patterns, and makes the search of patterns faster. In addition, a self-interacting network does not show a complete loss of memory at any stage. Instead, its memory degrades slowly so that retrieved patterns become progressively more and more erroneous. The relative weight of self-interaction is varied to determine its optimum amount in an N = 100 network. This weight is then used to study the retrieval characteristics of a network of 3-states sites for N = 50. It is found that a self-interacting network 3-states sites is more efficient than the usual binary network.

Analysis of the equilibrium states of the hopfield associative memory using the system of walsh functions

AbstractThis paper considers the situation where the multiple state‐patterns satisfying the orthogonality condition are memorized in the Hopfield associative memory model. Also discussed is the relation between the equilibrium state produced in the associative memory model and the memorized patterns. Numerous studies have been made of the associative memory model by the statistical approach. By contrast, this study considers the problem from the algebraic standpoint, using the Walsh functional system.As the first step, a sufficient condition is derived for the associative memory model to be in the equilibrium state. Using this condition, the equilibrium states are classified into three types and the relation between the memorized pattern and the generated equilibrium state is analyzed. As applications, a method of estimating the number of equilibrium states and their state pattern from the data concerning the memorized pattern, as well as a method of making the given pattern an equilibrium state, are shown through examples. the results are obtained by the component analysis of the state of the associative memory model using the Walsh functional system. In other words, the usefulness of applying the Walsh functional system is demonstrated.

Modeling of component failure in neural networks for robustness evaluation: an application to object extraction

The robustness of neural network (NN) based information processing systems with respect to component failure (damaging of nodes/links) is studied. The damaging/component failure process has been modeled as a Poisson process. To choose the instants or moments of damaging, statistical sampling technique is used. The nodes/links to be damaged are determined randomly. As an illustration, the model is implemented and tested on different object extraction algorithms employing Hopfield's associative memory model, Gibbs random fields, and a self-organizing multilayer neural network. The performance of these algorithms is evaluated in terms of percentage of pixels correctly classified under different noisy environments and different degrees and sequences of damaging. The deterioration in the output is seen to be very small even when a large number of nodes/links are damaged.

Theory of the dynamics of the Hopfield model of associative memory

We present an analysis of the parallel dynamics of the Hopfield model of the associative memory of a neural network without recourse to the replica formalism. A probabilistic method based on the signal-to-noise ratio is employed to obtain a simple recursion relation for the zero temperature as well as the finite temperature dynamics of the network. The fixed points of the recursion relation and their basins of attraction are found to be in fairly satisfactory agreement with the numerical simulations of the model. We also present some new numerical results which support our recursion relation and throw light on the nature of the ensemble of the network states which are optimized with respect to single spin flips.

MIXTURE STATES AND STORAGE WITH CORRELATED PATTERNS IN HOPFIELD'S MODEL

The Hopfield model of associative memory with the Hebb learning rule is studied for a finite number p of correlated patterns. The storage capacity α = P/N is considered in a network with further P − p embedded uncorrelated patterns, and the corresponding phase diagrams are exhibited. Numerical simulations are carried out to discuss the retrieval quality and the basins of attraction of the network.