Capacity Of Associative Memory Research Articles

NEURAL NETWORK-BASED METHODS FOR FINDING THE SHORTEST PATH and establishing associative connections between objects

Nowadays, solving optimizations problems is one of the tasks for intelligent computer systems. Currently, there is a problem of insufficient efficiency of optimizations tasks solving methods (for example, high computing time and/or accuracy). The object of the research is the process of finding the shortest path and establishing associative connections between objects. The subject of the research is the methods of finding the shortest path and establishing associative connections between objects based on neural networks with associative memory and neural network reinforcement training. The objective of this work is to improve the efficiency of finding the shortest path and establishing associative connections between objects through neural networks with associative memory and neural network reinforcement training. To achieve this goal, a neuro-associative method and a neural network reinforcement training method was developed. The advantages of the proposed methods include the following. First, the proposed bi-directional recurrent correlative associative memory, which uses hetero-associative and auto-associative memory and an exponential weighting function, allows for increasing the associative memory capacity while preserving learning accuracy. Second, the Deep Q-Network (DQN) reinforcement learning method with dynamic parameters uses the ε-greedy approach, which in the initial iterations is close to random search, and in the final iterations is close to directed search, which is ensured by using dynamic parameters and allows increasing the learning speed while preserving learning accuracy. Conducted numerical research allowed us to estimate both methods (for the first method, the root mean square error was 0.02, and for the second method it was 0.05). The proposed methods allow expanding the field of application of neural networks with associative memory and neural network reinforcement learning, which is confirmed by their adaptation for the tasks of finding the shortest path and establishing associative connections between objects and contribute to the effectiveness of intelligent computer systems of general and special purpose. Prospects for further research are to investigate the proposed methods for a wide class of artificial intelligence problems.

Classification of handwritten digits using the Hopfield network

The paper presents the results of the classification of handwritten digits from the MNIST database using the Hopfield network. A strong correlation of training binary patterns does not allow the use of the standard Hebbian learning method. The application of the Storkey learning method increases the capacity of associative memory, and the optimized pattern binarization threshold and pattern size reduce the correlation of patterns. By optimizing these parameters, a network achieved a classification accuracy of 56.2% on a set of validation data used for network training. The selection of the optimal binarization threshold for a separate set of test images increased the classification accuracy to 61.5%.

Optimality of sparse olfactory representations is not affected by network plasticity.

The neural representation of a stimulus is repeatedly transformed as it moves from the sensory periphery to deeper layers of the nervous system. Sparsening transformations are thought to increase the separation between similar representations, encode stimuli with great specificity, maximize storage capacity of associative memories, and provide an energy efficient instantiation of information in neural circuits. In the insect olfactory system, odors are initially represented in the periphery as a combinatorial code with relatively simple temporal dynamics. Subsequently, in the antennal lobe this representation is transformed into a dense and complex spatiotemporal activity pattern. Next, in the mushroom body Kenyon cells (KCs), the representation is dramatically sparsened. Finally, in mushroom body output neurons (MBONs), the representation takes on a new dense spatiotemporal format. Here, we develop a computational model to simulate this chain of olfactory processing from the receptor neurons to MBONs. We demonstrate that representations of similar odorants are maximally separated, measured by the distance between the corresponding MBON activity vectors, when KC responses are sparse. Sparseness is maintained across variations in odor concentration by adjusting the feedback inhibition that KCs receive from an inhibitory neuron, the Giant GABAergic neuron. Different odor concentrations require different strength and timing of feedback inhibition for optimal processing. Importantly, as observed in vivo, the KC–MBON synapse is highly plastic, and, therefore, changes in synaptic strength after learning can change the balance of excitation and inhibition, potentially leading to changes in the distance between MBON activity vectors of two odorants for the same level of KC population sparseness. Thus, what is an optimal degree of sparseness before odor learning, could be rendered sub–optimal post learning. Here, we show, however, that synaptic weight changes caused by spike timing dependent plasticity increase the distance between the odor representations from the perspective of MBONs. A level of sparseness that was optimal before learning remains optimal post-learning.

Singular value Decomposition applied to Associative Memory of Hopfield Neural Network

Content-addressable memories are useful for storage and data retrieval from arrays of sensors. The Hopfield neural network is a model of associative content-addressable memory with a simple flexible structure. Design of this artificial neural network is capable of memorizing large quantity of data and recalling the same from available information. In this work, we have shown that with corrupted input dataset, the correct set of data can be retrieved from approximated or compressed associative memory matrix. The idea has been explained through an experiment using binary datasets. The proposed methodology will increase the storage capacity of associative memory. In the experiment we have shown that it will minimize the number of arithmetic operations involved in recovery process of the Hopfield model but will also ensure correctness of information retrieval from the synaptic matrix from a corrupted dataset as input.

Paternal spatial training enhances offspring\u2019s cognitive performance and synaptic plasticity in wild-type but not improve memory deficit in Alzheimer\u2019s mice

Recent studies suggest that spatial training can maintain associative memory capacity in Tg2576 mice, but it is not known whether the beneficial effects can be inherited from the trained fathers to their offspring. Here, we exposed male wild-type and male 3XTg Alzheimer disease (AD) mice (3-m old) respectively to spatial training for one week and assessed the transgenerational effects in the F1 offspring when they were grown to 7-m old. We found that the paternal spatial training significantly enhanced progeny’s spatial cognitive performance and synaptic transmission in wild-type mice. Among several synapse- or memory-associated proteins, we observed that the expression level of synaptotagmin 1 (SYT1) was significantly increased in the hippocampus of the paternally trained-offspring. Paternal training increased histone acetylation at the promoter of SYT1 in both fathers’ and the offspring’s hippocampus, and as well as in the fathers’ sperm. Finally, paternal spatial training for one week did not improve memory and synaptic plasticity in 3XTg AD F1 offspring. Our findings suggest paternal spatial training for one week benefits the offspring’s cognitive performance in wild-type mice with the mechanisms involving an enhanced transgenerational histone acetylation at SYT1 promoter.

Pseudo-Orthogonalization of Memory Patterns for Complex-Valued and Quaternionic Associative Memories

Abstract Hebbian learning rule is well known as a memory storing scheme for associative memory models. This scheme is simple and fast, however, its performance gets decreased when memory patterns are not orthogonal each other. Pseudo-orthogonalization is a decorrelating method for memory patterns which uses XNOR masking between the memory patterns and randomly generated patterns. By a combination of this method and Hebbian learning rule, storage capacity of associative memory concerning non-orthogonal patterns is improved without high computational cost. The memory patterns can also be retrieved based on a simulated annealing method by using an external stimulus pattern. By utilizing complex numbers and quaternions, we can extend the pseudo-orthogonalization for complex-valued and quaternionic Hopfield neural networks. In this paper, the extended pseudo-orthogonalization methods for associative memories based on complex numbers and quaternions are examined from the viewpoint of correlations in memory patterns. We show that the method has stable recall performance on highly correlated memory patterns compared to the conventional real-valued method.

Spatial training preserves associative memory capacity with augmentation of dendrite ramification and spine generation in Tg2576 mice.

Alzheimer's disease (AD) is the most common neurodegenerative disorder and there is currently no efficient cure for this devastating disease. Cognitive stimulation can delay memory loss during aging and in patients with mild cognitive impairment. In 3 × Tg-AD mice, training decreased the neuropathologies with transient amelioration of memory decline. However, the neurobiological mechanisms underlying the learning-improved memory capacity are poorly understood. Here, we found in Tg2576 mice spatial training in Morris water maze (MWM) remarkably improved the subsequent associative memory acquisition detected by contextual fear conditioning. We also found that spatial training enhanced long term potentiation, dendrite ramification and spine generation in hippocampal dentate gyrus (DG) and CA1 neurons at 24 h after the training. In the molecular level, the MWM training remarkably activated calcium/calmodulin-dependent protein kinase II (CaMKII) with elevation of glutamate AMPA receptor GluA1 subunit (GluA1), postsynaptic density protein 93 (PSD93) and postsynaptic density protein 95 (PSD95) in the hippocampus. Finally, the training also significantly ameliorated AD-like tau and amyloid pathologies. We conclude that spatial training in MWM preserves associative memory capacity in Tg2576 mice, and the mechanisms involve augmentation of dendrite ramification and spine generation in hippocampus.

Evaluation Model of Bridges Health State Based on DHNN

Inthis paper, according to the more important ten evaluation indicators, the fourgrades ideal evaluation is established corresponding to the level of healthstate of bridges. Combined with associative memory capacity of discreteHopfield neural networks, a new health state evaluation of bridges ispresented. Five bridges is evaluated by the model, the network connectionweights is obtained by iterative learning using the outer product method. Thesimulation results shows that the health evaluation model can evaluate thehealth state of bridges fast, accurately and intuitively.

Effect of Ppropofol on spatial associative memory capacity in rats using morris water maze

Objective To study the clinical effects of low-dose and high-dose injection of propefol on spatial associative memory capacity in rats. Methods 30 rats were randomly divided into 3 groups: control, clin-ical low-dose propofol (0.1 mg/kg·min) group and clinical high-dose propofol (0.5 mg/kg·min) group. The experi-ments including 4-days place swimming and 1-day space exploration began from the day after injection. In place swimming group, rats were trained in morris water maze from Ⅰ,Ⅱ,Ⅲ quadrant. Latency period, swimming length and swimming velocity were recorded. In space exploration group, untrained rats were put into the maze from the Ⅳ quadrant and swimming time, first circling point value and total scores were recorded. Results At the first two days of place swimming test there were no differences between the control group and the low-dose group in la-tency, swimming length and swimming velocity(P<0.05); while high-dose group's latency period and swimming length were obviously longer than that of control(P<0.05). At the second two days there were no differences among the three groups(P<0.05). In the space exploration test, total scores of control, low-dose group and high-dose group were: 2 509.36±190.72, 2 378.55±210.69 and 1 954.85±174.20, respectively. There was no difference be-tween low-dose group and the control group(P<0.05). Total scores of the high-dose group were much lower than that of control. Conclusion The clinical low-dose propofol injection has no effects on the spatial associative memory capacity and spatial memory capacity of rats. The clinical high-dose propofol injection affects the rats' spatial memory capacity during the first two days but has no effects after two days, while the clinical high-dose propofol injection reduces the rats' spatial associative memory capacity. Key words: Propofol; Rats; Spatial associative memory capacity; Morris water maze; Spatial memory capacity

Long-term associative memory capacity in man

Long-term episodic memory depends on the retention of associative information, such as the relationship between a friend's face and his name, a home and its neighborhood, and a mint and its odor. How many of these associative links can be stored and recalled? Many have considered memory's capacity, but relevant data are scant. Some previous experiments on humans have assessed the retention of thousands of visual impressions using forced-choice picture recognition and have concluded that there are virtually no constraints on how much information can be successfully retained. However, no previous experiments on humans have investigated the capacity of associative memory. I describe the first relevant data, which I obtained by systematically probing my own capacity during 58,560 memory trials for picture-response associations (approximately 1 year of testing). Estimated capacity was on the order of several thousand associations, and this and other indicators of memory function were remarkably similar to those obtained for baboons (Papio papio) under comparable circumstances. These findings, along with other data, suggest conservation of long-term memory mechanisms and effectiveness in humans relative to nonhuman primates, despite at least 20 million years of divergent evolution and vastly different behavioral and cognitive repertoires. The present findings also indicate that the associative processes that support our ability to remember episodes are limited in capacity relative to processes that support picture recognition.

Quantum learning for neural associative memories

Quantum information processing in neural structures results in an exponential increase of patterns storage capacity and can explain the extensive memorization and inferencing capabilities of humans. An example can be found in neural associative memories if the synaptic weights are taken to be fuzzy variables. In that case, the weights’ update is carried out with the use of a fuzzy learning algorithm which satisfies basic postulates of quantum mechanics. The resulting weight matrix can be decomposed into a superposition of associative memories. Thus, the fundamental memory patterns (attractors) can be mapped into different vector spaces which are related to each other via unitary rotations. Quantum learning increases the storage capacity of associative memories by a factor of 2 N , where N is the number of neurons.

The restricted influence of sparseness of coding on the capacity of familiarity discrimination networks

Much evidence indicates that the perirhinal cortex is involved in the familiarity discrimination aspect of recognition memory. It has been previously shown under selective conditions that neural networks performing familiarity discrimination can achieve very high storage capacity, being able to deal with many times more stimuli than associative memory networks can in associative recall. The capacity of associative memories for recall has been shown to be highly dependent on the sparseness of coding. However, previous work on the networks of Bogacz et al, Norman and O'Reilly and Sohal and Hasselmo that model familiarity discrimination in the perirhinal cortex has not investigated the effects of the sparseness of encoding on capacity. This paper explores how sparseness of coding influences the capacity of each of these published models and establishes that sparse coding influences the capacity of the different models in different ways. The capacity of the Bogacz et al model can be made independent of the sparseness of coding. Capacity increases as coding becomes sparser for a simplified version of the neocortical part of the Norman and O'Reilly model, whereas capacity decreases as coding becomes sparser for a simplified version of the Sohal and Hasselmo model. Thus in general, and in contrast to associative memory networks, sparse encoding results in little or no advantage for the capacity of familiarity discrimination networks. Hence it may be less important for coding to be sparse in the perirhinal cortex than it is in the hippocampus. Additionally, it is established that the capacities of the networks are strongly dependent on the precise form of the learning rules (synaptic plasticity) used in the network. This finding indicates that the precise characteristics of synaptic plastic changes in the real brain are likely to have major influences on storage capacity.

The restricted influence of sparseness of coding on the capacity of familiarity discrimination networks

Much evidence indicates that the perirhinal cortex is involved in the familiarity discrimination aspect of recognition memory. It has been previously shown under selective conditions that neural networks performing familiarity discrimination can achieve very high storage capacity, being able to deal with many times more stimuli than associative memory networks can in associative recall. The capacity of associative memories for recall has been shown to be highly dependent on the sparseness of coding. However, previous work on the networks of Bogacz et al, Norman and O'Reilly and Sohal and Hasselmo that model familiarity discrimination in the perirhinal cortex has not investigated the effects of the sparseness of encoding on capacity. This paper explores how sparseness of coding influences the capacity of each of these published models and establishes that sparse coding influences the capacity of the different models in different ways. The capacity of the Bogacz et al model can be made independent of the sparseness of coding. Capacity increases as coding becomes sparser for a simplified version of the neocortical part of the Norman and O'Reilly model, whereas capacity decreases as coding becomes sparser for a simplified version of the Sohal and Hasselmo model. Thus in general, and in contrast to associative memory networks, sparse encoding results in little or no advantage for the capacity of familiarity discrimination networks. Hence it may be less important for coding to be sparse in the perirhinal cortex than it is in the hippocampus. Additionally, it is established that the capacities of the networks are strongly dependent on the precise form of the learning rules (synaptic plasticity) used in the network. This finding indicates that the precise characteristics of synaptic plastic changes in the real brain are likely to have major influences on storage capacity.

Estimating the storage capacity of space-varyingcellular neural networks

The capacity of associative memories based on space-varying, nonreciprocal, discrete-time cellular neural networks (CNNs) is investigated. The capacity is defined as the maximum number of random bipolar prototypes which can be stored with probability larger than a suitable threshold. It is shown that the capacity of the CNN is actually determined by the cells with the smallest number of connections, i.e. those located at the corners of the rectangular array. A simple empirical formula is presented which enables a prediction to be made as to the capacity of the CNN associative memory as a function of the neighbourhood radius and of the stability margin.

Self-consistent signal-to-noise analysis of the statistical behavior of analog neural networks and enhancement of the storage capacity.

Based on the self-consistent signal-to-noise analysis (SCSNA) capable of dealing with analog neural networks with a wide class of transfer functions, enhancement of the storage capacity of associative memory and the related statistical properties of neural networks are studied for random memory patterns. Two types of transfer functions with the threshold parameter \ensuremath{\theta} are considered, which are derived from the sigmoidal one to represent the output of three-state neurons. Neural networks having a monotonically increasing transfer function ${\mathit{F}}^{\mathrm{M}}$, ${\mathit{F}}^{\mathrm{M}}$(u)=sgnu (\ensuremath{\Vert}u\ensuremath{\Vert}g\ensuremath{\theta}), ${\mathit{F}}^{\mathrm{M}}$(u)=0 (\ensuremath{\Vert}u\ensuremath{\Vert}\ensuremath{\le}\ensuremath{\theta}), are shown to make it impossible for the spin-glass state to coexist with retrieval states in a certain parameter region of \ensuremath{\theta} and \ensuremath{\alpha} (loading rate of memory patterns), implying the reduction of the number of spurious states. The behavior of the storage capacity with changing \ensuremath{\theta} is qualitatively the same as that of the Ising spin neural networks with varying temperature. On the other hand, the nonmonotonic transfer function ${\mathit{F}}^{\mathrm{NM}}$, ${\mathit{F}}^{\mathrm{NM}}$(u)=sgnu (\ensuremath{\Vert}u\ensuremath{\Vert}), ${\mathit{F}}^{\mathrm{NM}}$(u)=0 (\ensuremath{\Vert}u\ensuremath{\Vert}\ensuremath{\ge}\ensuremath{\theta}) gives rise to remarkable features in several respects.First, it yields a large enhancement of the storage capacity compared with the Amit-Gutfreund-Sompolinsky (AGS) value: with decreasing \ensuremath{\theta} from \ensuremath{\theta}=\ensuremath{\infty}, the storage capacity ${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$ of such a network is increased from the AGS value (\ensuremath{\approxeq}0.14) to attain its maximum value of \ensuremath{\approxeq}0.42 at \ensuremath{\theta}\ensuremath{\simeq}0.7 and afterwards is decreased to vanish at \ensuremath{\theta}=0. Whereas for \ensuremath{\theta}\ensuremath{\gtrsim}1 the storage capacity ${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$ coincides with the value ${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$\ifmmode \tilde{}\else \~{}\fi{} determined by the SCSNA as the upper bound of \ensuremath{\alpha} ensuring the existence of retrieval solutions, for \ensuremath{\theta}\ensuremath{\lesssim}1 the ${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$ is shown to differ from the ${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$\ifmmode \tilde{}\else \~{}\fi{} with the result that the retrieval solutions claimed by the SCSNA are unstable for ${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$\ifmmode \tilde{}\else \~{}\fi{}. Second, in the case of \ensuremath{\theta}1 the network can exhibit a new type of phase which appears as a result of a phase transition with respect to the non-Gaussian distribution of the local fields of neurons: the standard type of retrieval state with r\ensuremath{\ne}0 (i.e., finite width of the local field distribution), which is implied by the order-parameter equations of the SCSNA, disappears at a certain critical loading rate ${\mathrm{\ensuremath{\alpha}}}_{0}$, and for \ensuremath{\alpha}\ensuremath{\le}${\mathrm{\ensuremath{\alpha}}}_{0}$ a qualitatively different type of retrieval state comes into existence in which the width of the local field distribution vanishes (i.e., r=${0}^{+}$). As a consequence, memory retrieval without errors becomes possible even in the saturation limit \ensuremath{\alpha}\ensuremath{\ne}0. Results of the computer simulations on the statistical properties of the novel phase with \ensuremath{\alpha}\ensuremath{\le}${\mathrm{\ensuremath{\alpha}}}_{0}$ are shown to be in satisfactory agreement with the theoretical results. The effect of introducing self-couplings on the storage capacity is also analyzed for the two types of networks. It is conspicuous for the networks with ${\mathit{F}}^{\mathrm{NM}}$, where the self-couplings increase the stability of the retrieval solutions of the SCSNA with small values of \ensuremath{\theta}, leading to a remarkable enhancement of the storage capacity.

The capacity of associative memories with malfunctioning neurons

Hopfield associative memories with alphan malfunctioning neurons are considered. Using some facts from exchangeable events theory, the asymptotic storage capacity of such a network is derived as a function of the parameter alpha under stability and attractivity requirements. It is shown that the asymptotic storage capacity is (1-alpha)(2)/n(4 log n) under stability and (1-alpha)(2)(1-2rho)(2)n/(4 log n) under attractivity requirements, respectively. Comparing these capacities with their maximum values corresponding to the case when there is no malfunctioning neurons, alpha=0, shows the robustness of the retrieval mechanism of Hopfield associative memories with respect to the existence of malfunctioning neurons. This result also supports the claim that neural networks are fault tolerant.

Capacity of associative memory using a nonmonotonic neuron model

Associative memory using a sigmoid neuron model with the autocorrelation matrix has the advantage of the simplicity of its structure of the memory but has the disadvantage of the memory capacity. Its absolute capacity is asymptotically n (2 log n) , where n is the number of neurons. By computer simulation, Morita has recently shown that the performance of the associative memory is improved remarkably by replacing the usual sigmoid neuron with a nonmonotonic one, without sacrificing the simplicity. We use a piecewise linear model of the nonmonotonic neuron and investigate the existence and stability of equilibrium states of the recalling process. We derive two kinds of theoretical estimates of the absolute capacity. One estimate gives the upper bound of the absolute capacity 0.5n, and the other gives the average value of the absolute capacity 0.4n. These values fit well with computer simulations.

Recurrent correlation associative memories

A model for a class of high-capacity associative memories is presented. Since they are based on two-layer recurrent neural networks and their operations depend on the correlation measure, these associative memories are called recurrent correlation associative memories (RCAMs). The RCAMs are shown to be asymptotically stable in both synchronous and asynchronous (sequential) update modes as long as their weighting functions are continuous and monotone nondecreasing. In particular, a high-capacity RCAM named the exponential correlation associative memory (ECAM) is proposed. The asymptotic storage capacity of the ECAM scales exponentially with the length of memory patterns, and it meets the ultimate upper bound for the capacity of associative memories. The asymptotic storage capacity of the ECAM with limited dynamic range in its exponentiation nodes is found to be proportional to that dynamic range. Design and fabrication of a 3-mm CMOS ECAM chip is reported. The prototype chip can store 32 24-bit memory patterns, and its speed is higher than one associative recall operation every 3 mus. An application of the ECAM chip to vector quantization is also described.

On the capacity of associative memories with linear threshold functions

Some important features of various known constructions of associative memories based on linear threshold functions are analyzed. Two important features are dealt with: (a) the ability to select an arbitrary set of desired memory vectors and design a network for this set; (b) the sizes and shapes of the domains of attraction of the desired memory vectors and their relation to various design parameters. The static capacity for randomly chosen desired memories is also analyzed. Two extremal examples of sets of desired memories are then analyzed in detail. For spectral schemes with randomly chosen O(N/ln N) memories, it is shown that almost all of the Hamming sphere around each memory is directly attracted.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Information capacity of associative memories

Associative memory networks consisting of highly interconnected binary-valued cells have been used to model neural networks. Tight asymptotic bounds have been found for the information capacity of these networks. The authors derive the asymptotic information capacity of these networks using results from normal approximation theory and theorems about exchangeable random variables. >