Storage Capacity Of Associative Memory Research Articles

Optimality of sparse olfactory representations is not affected by network plasticity.

The neural representation of a stimulus is repeatedly transformed as it moves from the sensory periphery to deeper layers of the nervous system. Sparsening transformations are thought to increase the separation between similar representations, encode stimuli with great specificity, maximize storage capacity of associative memories, and provide an energy efficient instantiation of information in neural circuits. In the insect olfactory system, odors are initially represented in the periphery as a combinatorial code with relatively simple temporal dynamics. Subsequently, in the antennal lobe this representation is transformed into a dense and complex spatiotemporal activity pattern. Next, in the mushroom body Kenyon cells (KCs), the representation is dramatically sparsened. Finally, in mushroom body output neurons (MBONs), the representation takes on a new dense spatiotemporal format. Here, we develop a computational model to simulate this chain of olfactory processing from the receptor neurons to MBONs. We demonstrate that representations of similar odorants are maximally separated, measured by the distance between the corresponding MBON activity vectors, when KC responses are sparse. Sparseness is maintained across variations in odor concentration by adjusting the feedback inhibition that KCs receive from an inhibitory neuron, the Giant GABAergic neuron. Different odor concentrations require different strength and timing of feedback inhibition for optimal processing. Importantly, as observed in vivo, the KC–MBON synapse is highly plastic, and, therefore, changes in synaptic strength after learning can change the balance of excitation and inhibition, potentially leading to changes in the distance between MBON activity vectors of two odorants for the same level of KC population sparseness. Thus, what is an optimal degree of sparseness before odor learning, could be rendered sub–optimal post learning. Here, we show, however, that synaptic weight changes caused by spike timing dependent plasticity increase the distance between the odor representations from the perspective of MBONs. A level of sparseness that was optimal before learning remains optimal post-learning.

Singular value Decomposition applied to Associative Memory of Hopfield Neural Network

Content-addressable memories are useful for storage and data retrieval from arrays of sensors. The Hopfield neural network is a model of associative content-addressable memory with a simple flexible structure. Design of this artificial neural network is capable of memorizing large quantity of data and recalling the same from available information. In this work, we have shown that with corrupted input dataset, the correct set of data can be retrieved from approximated or compressed associative memory matrix. The idea has been explained through an experiment using binary datasets. The proposed methodology will increase the storage capacity of associative memory. In the experiment we have shown that it will minimize the number of arithmetic operations involved in recovery process of the Hopfield model but will also ensure correctness of information retrieval from the synaptic matrix from a corrupted dataset as input.

Pseudo-Orthogonalization of Memory Patterns for Complex-Valued and Quaternionic Associative Memories

Abstract Hebbian learning rule is well known as a memory storing scheme for associative memory models. This scheme is simple and fast, however, its performance gets decreased when memory patterns are not orthogonal each other. Pseudo-orthogonalization is a decorrelating method for memory patterns which uses XNOR masking between the memory patterns and randomly generated patterns. By a combination of this method and Hebbian learning rule, storage capacity of associative memory concerning non-orthogonal patterns is improved without high computational cost. The memory patterns can also be retrieved based on a simulated annealing method by using an external stimulus pattern. By utilizing complex numbers and quaternions, we can extend the pseudo-orthogonalization for complex-valued and quaternionic Hopfield neural networks. In this paper, the extended pseudo-orthogonalization methods for associative memories based on complex numbers and quaternions are examined from the viewpoint of correlations in memory patterns. We show that the method has stable recall performance on highly correlated memory patterns compared to the conventional real-valued method.

Quantum learning for neural associative memories

Quantum information processing in neural structures results in an exponential increase of patterns storage capacity and can explain the extensive memorization and inferencing capabilities of humans. An example can be found in neural associative memories if the synaptic weights are taken to be fuzzy variables. In that case, the weights’ update is carried out with the use of a fuzzy learning algorithm which satisfies basic postulates of quantum mechanics. The resulting weight matrix can be decomposed into a superposition of associative memories. Thus, the fundamental memory patterns (attractors) can be mapped into different vector spaces which are related to each other via unitary rotations. Quantum learning increases the storage capacity of associative memories by a factor of 2 N , where N is the number of neurons.

Self-consistent signal-to-noise analysis of the statistical behavior of analog neural networks and enhancement of the storage capacity.

Based on the self-consistent signal-to-noise analysis (SCSNA) capable of dealing with analog neural networks with a wide class of transfer functions, enhancement of the storage capacity of associative memory and the related statistical properties of neural networks are studied for random memory patterns. Two types of transfer functions with the threshold parameter \ensuremath{\theta} are considered, which are derived from the sigmoidal one to represent the output of three-state neurons. Neural networks having a monotonically increasing transfer function ${\mathit{F}}^{\mathrm{M}}$, ${\mathit{F}}^{\mathrm{M}}$(u)=sgnu (\ensuremath{\Vert}u\ensuremath{\Vert}g\ensuremath{\theta}), ${\mathit{F}}^{\mathrm{M}}$(u)=0 (\ensuremath{\Vert}u\ensuremath{\Vert}\ensuremath{\le}\ensuremath{\theta}), are shown to make it impossible for the spin-glass state to coexist with retrieval states in a certain parameter region of \ensuremath{\theta} and \ensuremath{\alpha} (loading rate of memory patterns), implying the reduction of the number of spurious states. The behavior of the storage capacity with changing \ensuremath{\theta} is qualitatively the same as that of the Ising spin neural networks with varying temperature. On the other hand, the nonmonotonic transfer function ${\mathit{F}}^{\mathrm{NM}}$, ${\mathit{F}}^{\mathrm{NM}}$(u)=sgnu (\ensuremath{\Vert}u\ensuremath{\Vert}), ${\mathit{F}}^{\mathrm{NM}}$(u)=0 (\ensuremath{\Vert}u\ensuremath{\Vert}\ensuremath{\ge}\ensuremath{\theta}) gives rise to remarkable features in several respects.First, it yields a large enhancement of the storage capacity compared with the Amit-Gutfreund-Sompolinsky (AGS) value: with decreasing \ensuremath{\theta} from \ensuremath{\theta}=\ensuremath{\infty}, the storage capacity ${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$ of such a network is increased from the AGS value (\ensuremath{\approxeq}0.14) to attain its maximum value of \ensuremath{\approxeq}0.42 at \ensuremath{\theta}\ensuremath{\simeq}0.7 and afterwards is decreased to vanish at \ensuremath{\theta}=0. Whereas for \ensuremath{\theta}\ensuremath{\gtrsim}1 the storage capacity ${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$ coincides with the value ${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$\ifmmode \tilde{}\else \~{}\fi{} determined by the SCSNA as the upper bound of \ensuremath{\alpha} ensuring the existence of retrieval solutions, for \ensuremath{\theta}\ensuremath{\lesssim}1 the ${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$ is shown to differ from the ${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$\ifmmode \tilde{}\else \~{}\fi{} with the result that the retrieval solutions claimed by the SCSNA are unstable for ${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$\ifmmode \tilde{}\else \~{}\fi{}. Second, in the case of \ensuremath{\theta}1 the network can exhibit a new type of phase which appears as a result of a phase transition with respect to the non-Gaussian distribution of the local fields of neurons: the standard type of retrieval state with r\ensuremath{\ne}0 (i.e., finite width of the local field distribution), which is implied by the order-parameter equations of the SCSNA, disappears at a certain critical loading rate ${\mathrm{\ensuremath{\alpha}}}_{0}$, and for \ensuremath{\alpha}\ensuremath{\le}${\mathrm{\ensuremath{\alpha}}}_{0}$ a qualitatively different type of retrieval state comes into existence in which the width of the local field distribution vanishes (i.e., r=${0}^{+}$). As a consequence, memory retrieval without errors becomes possible even in the saturation limit \ensuremath{\alpha}\ensuremath{\ne}0. Results of the computer simulations on the statistical properties of the novel phase with \ensuremath{\alpha}\ensuremath{\le}${\mathrm{\ensuremath{\alpha}}}_{0}$ are shown to be in satisfactory agreement with the theoretical results. The effect of introducing self-couplings on the storage capacity is also analyzed for the two types of networks. It is conspicuous for the networks with ${\mathit{F}}^{\mathrm{NM}}$, where the self-couplings increase the stability of the retrieval solutions of the SCSNA with small values of \ensuremath{\theta}, leading to a remarkable enhancement of the storage capacity.