Neural Associative Memory Research Articles

Full CMOS Circuit for Brain-Inspired Associative Memory With On-Chip Trainable Memristive STDP Synapse

Spiking neural networks (SNNs) implemented in neuromorphic computing architectures promise a high degree of bio-plausibility and energy efficiency compared to the artificial neural network (ANN). Thus, SNN-based spiking associative memories are preferred for high capacity, area, and energy-efficient neural associative memories (NAMs). While most previously published works focused on ANN-based NAM, this work implements the full CMOS circuit of memristor crossbar-based spiking NAM for the first time. Instead of using any software-based memristive SPICE model or memristive devices that are yet not available in standard CMOS technology process design kits (PDKs), in our work, the CMOS-based memristive synapse circuit is employed to address practical circuit implementation challenges. The complete on-chip learning of the system is demonstrated using the bio-plausible spike-timing-dependent plasticity (STDP) learning mechanism without employing any external coprocessor, e.g., microprocessor, field-programmable gate array (FPGA). The entire system is implemented at the transistor level using 180-nm standard CMOS technology to demonstrate the pattern recognition application. The robustness of the proposed circuit is also evaluated to demonstrate the tolerance against the CMOS fabrication non-idealities.

On Neural Associative Memory Structures: Storage and Retrieval of Sequences in a Chain of Tournaments.

Associative memories enjoy many interesting properties in terms of error correction capabilities, robustness to noise, storage capacity, and retrieval performance, and their usage spans over a large set of applications. In this letter, we investigate and extend tournament-based neural networks, originally proposed by Jiang, Gripon, Berrou, and Rabbat (2016), a novel sequence storage associative memory architecture with high memory efficiency and accurate sequence retrieval. We propose a more general method for learning the sequences, which we call feedback tournament-based neural networks. The retrieval process is also extended to both directions: forward and backward-in other words, any large-enough segment of a sequence can produce the whole sequence. Furthermore, two retrieval algorithms, cache-winner and explore-winner, are introduced to increase the retrieval performance. Through simulation results, we shed light on the strengths and weaknesses of each algorithm.

Noise Robust Projection Rule for Klein Hopfield Neural Networks

Multistate Hopfield models, such as complex-valued Hopfield neural networks (CHNNs), have been used as multistate neural associative memories. Quaternion-valued Hopfield neural networks (QHNNs) reduce the number of weight parameters of CHNNs. The CHNNs and QHNNs have weak noise tolerance by the inherent property of rotational invariance. Klein Hopfield neural networks (KHNNs) improve the noise tolerance by resolving rotational invariance. However, the KHNNs have another disadvantage of self-feedback, a major factor of deterioration in noise tolerance. In this work, the stability conditions of KHNNs are extended. Moreover, the projection rule for KHNNs is modified using the extended conditions. The proposed projection rule improves the noise tolerance by a reduction in self-feedback. Computer simulations support that the proposed projection rule improves the noise tolerance of KHNNs.

Two-Level Complex-Valued Hopfield Neural Networks.

In multistate neural associative memories, some neurons have small noise and the others have large noise. If we know which neurons have small noise, the noise tolerance could be improved. In this brief, we provide a novel method to reinforce neurons with small noise and apply our new method to images with the Gaussian noise. A complex-valued multistate neuron is decomposed to two neurons, referred to as high and low neurons. For the Gaussian noise, the high neurons are expected to have small noise. The noise tolerance is improved by reinforcement of high neurons. The computer simulations support the efficiency of reinforced neurons.

Hyperbolic-valued Hopfield neural networks in hybrid mode

A complex-valued Hopfield neural network (CHNN) has been applied as a multistate neural associative memory. Although synchronous mode accelerates the recall process, a CHNN with a projection rule may be trapped at a cycle of length two. A hyperbolic-valued Hopfield neural network (HHNN) with a conventional projection rule converges to a fixed point in synchronous mode. A noise robust projection rule improves the noise tolerance of HHNN, though it is not able to be applied to an HHNN in synchronous mode. In this work, we proposed hybrid mode, that is, asynchronous mode after synchronous mode. The conventional projection rule is employed in synchronous mode, and the noise robust projection rule is employed in asynchronous mode. Thus, the HHNN in hybrid mode converges in both modes. The HHNN in hybrid mode is expected to provide better noise tolerance than an HHNN in synchronous mode and faster recall than an HHNN in asynchronous mode. Computer simulations imply that our expectations are achieved.

Stability Conditions of Bicomplex-Valued Hopfield Neural Networks.

Hopfield neural networks have been extended using hypercomplex numbers. The algebra of bicomplex numbers, also referred to as commutative quaternions, is a number system of dimension 4. Since the multiplication is commutative, many notions and theories of linear algebra, such as determinant, are available, unlike quaternions. A bicomplex-valued Hopfield neural network (BHNN) has been proposed as a multistate neural associative memory. However, the stability conditions have been insufficient for the projection rule. In this work, the stability conditions are extended and applied to improvement of the projection rule. The computer simulations suggest improved noise tolerance.

Bicomplex-valued twin-hyperbolic Hopfield neural networks

A complex-valued multistate Hopfield neural network (CMHNN) is a multistate model of a Hopfield neural network and has been applied as a multistate neural associative memory. CMHNNs require many resources for weight parameters. To reduce the number of weight parameters, twin-multistate activation functions were introduced. A quaternion-valued twin-multistate Hopfield neural network (QTMHNN) is the first model to employ a twin-multistate activation function. A bicomplex-valued twin-multistate Hopfield neural network (BTMHNN) was also introduced. Weak noise tolerance is a disadvantage of QTMHNNs and BTMHNNs. To improve the noise tolerance, a BTMHNN can be modified to a bicomplex-valued twin–hyperbolic Hopfield neural network (BTHHNN). A BTMHNN is defined by the decomposition of a bicomplex number to a pair of complex numbers, whereas a BTHHNN is defined by decomposition of a bicomplex number to a pair of hyperbolic numbers. Computer simulations have improved the noise tolerance of BTHHNNs.

Bicomplex Projection Rule for Complex-Valued Hopfield Neural Networks.

A complex-valued Hopfield neural network (CHNN) with a multistate activation function is a multistate model of neural associative memory. The weight parameters need a lot of memory resources. Twin-multistate activation functions were introduced to quaternion- and bicomplex-valued Hopfield neural networks. Since their architectures are much more complicated than that of CHNN, the architecture should be simplified. In this work, the number of weight parameters is reduced by bicomplex projection rule for CHNNs, which is given by the decomposition of bicomplex-valued Hopfield neural networks. Computer simulations support that the noise tolerance of CHNN with a bicomplex projection rule is equal to or even better than that of quaternion- and bicomplex-valued Hopfield neural networks. By computer simulations, we find that the projection rule for hyperbolic-valued Hopfield neural networks in synchronous mode maintains a high noise tolerance.

Robust computation with rhythmic spike patterns

Information coding by precise timing of spikes can be faster and more energy efficient than traditional rate coding. However, spike-timing codes are often brittle, which has limited their use in theoretical neuroscience and computing applications. Here, we propose a type of attractor neural network in complex state space and show how it can be leveraged to construct spiking neural networks with robust computational properties through a phase-to-timing mapping. Building on Hebbian neural associative memories, like Hopfield networks, we first propose threshold phasor associative memory (TPAM) networks. Complex phasor patterns whose components can assume continuous-valued phase angles and binary magnitudes can be stored and retrieved as stable fixed points in the network dynamics. TPAM achieves high memory capacity when storing sparse phasor patterns, and we derive the energy function that governs its fixed-point attractor dynamics. Second, we construct 2 spiking neural networks to approximate the complex algebraic computations in TPAM, a reductionist model with resonate-and-fire neurons and a biologically plausible network of integrate-and-fire neurons with synaptic delays and recurrently connected inhibitory interneurons. The fixed points of TPAM correspond to stable periodic states of precisely timed spiking activity that are robust to perturbation. The link established between rhythmic firing patterns and complex attractor dynamics has implications for the interpretation of spike patterns seen in neuroscience and can serve as a framework for computation in emerging neuromorphic devices.

Recursive Synaptic Bit Reuse: An Efficient Way to Increase Memory Capacity in Associative Memory

Neural associative memory (AM) is one of the critical building blocks for cognitive computing systems. It memorizes (learns) and retrieves input data by information content itself. One of the key challenges of designing AM for intelligent devices is to expand memory capacity while using a minimal amount of hardware and energy resources. However, prior arts show that memory capacity increases slowly, i.e., in square root with the total number of synaptic weights. To tackle this problem, we propose a synapse model called recursive synaptic bit reuse, which enables near-linear scaling of memory capacity with total synaptic bits. Our model can also handle input data that are correlated more robustly than the conventional model. We evaluated our model in the context of Hopfield neural networks (HNNs) that contain 5–327-KB data storage for synaptic weights. Our model can increase the memory capacity of HNNs as large as $30\times $ over the conventional ones. The very large scale integration implementation of HNNs in 65 nm confirms that our proposed model can save up to $19\times $ area and up to $232\times $ energy dissipation as compared to the conventional model. These savings are expected to grow with the network size.

Binary Associative Memories as a Benchmark for Spiking Neuromorphic Hardware.

Large-scale neuromorphic hardware platforms, specialized computer systems for energy efficient simulation of spiking neural networks, are being developed around the world, for example as part of the European Human Brain Project (HBP). Due to conceptual differences, a universal performance analysis of these systems in terms of runtime, accuracy and energy efficiency is non-trivial, yet indispensable for further hard- and software development. In this paper we describe a scalable benchmark based on a spiking neural network implementation of the binary neural associative memory. We treat neuromorphic hardware and software simulators as black-boxes and execute exactly the same network description across all devices. Experiments on the HBP platforms under varying configurations of the associative memory show that the presented method allows to test the quality of the neuron model implementation, and to explain significant deviations from the expected reference output.

A sub-nJ CMOS ECG classifier for wireless smart sensor.

Body area sensor networks hold the promise of more efficient and cheaper medical care services through the constant monitoring of physiological markers such as heart beats. Continuously transmitting the electrocardiogram (ECG) signal requires most of the wireless ECG sensor energy budget. This paper presents the analog implantation of a classifier for ECG signals that can be embedded onto a sensor. The classifier is a sparse neural associative memory. It is implemented using the ST 65 nm CMOS technology and requires only 234 pJ per classification while achieving a 93.6% classification accuracy. The energy requirement is 6 orders of magnitude lower than a digital accelerator that performs a similar task. The lifespan of the resulting sensor is 191 times as large as that of a sensor sending all the data.

Nested-Clique Network Model of Neural Associative Memory.

Clique-based neural associative memories introduced by Gripon and Berrou (GB), have been shown to have good performance, and in our previous work we improved the learning capacity and retrieval rate by local coding and precoding in the presence of partial erasures. We now take a step forward and consider nested-clique graph structures for the network. The GB model stores patterns as small cliques, and we here replace these by nested cliques. Simulation results show that the nested-clique structure enhances the clique-based model.

Associative Memory Using Dictionary Learning and Expander Decoding

An associative memory is a framework of content-addressable memory that stores a collection of message vectors (or a dataset) over a neural network while enabling a neurally feasible mechanism to recover any message in the dataset from its noisy version. Designing an associative memory requires addressing two main tasks: 1) learning phase: given a dataset, learn a concise representation of the dataset in the form of a graphical model (or a neural network), 2) recall phase: given a noisy version of a message vector from the dataset, output the correct message vector via a neurally feasible algorithm over the network learnt during the learning phase. This paper studies the problem of designing a class of neural associative memories which learns a network representation for a large dataset that ensures correction against a large number of adversarial errors during the recall phase. Specifically, the associative memories designed in this paper can store dataset containing exp(n) n-length message vectors over a network with O(n) nodes and can tolerate Ω(n / polylog) adversarial errors. This paper carries out this memory design by mapping the learning phase and recall phase to the tasks of dictionary learning with a square dictionary and iterative error correction in an expander code, respectively.

The use of a multilayer neural network associative memory for optimal choice of process equipment

The use of a multilayer neural network associative memory for optimal choice of process equipment

Oscillatory neural associative memories with synapses based on memristor bridges

An approach to the implementation of electronic associative memories with tunable weights based on the resistor bridges containing memristors--a bidirectional associative memory (BAM) and an associative memory based on the Hopfield network--is proposed. These memories we implement as a networks of coupled phase oscillators. The conditions for the use of the operational amplifier in a comparator mode for implementing the step activation function are determined. It is shown how to use the CMOS transistors switches to control the memristance value. The experiments using LTSPICE models show that for the reference binary images with size 3 × 3 the proposed networks converges to the reference images (and, accordingly, to their inversion) with a random uniform distribution of binary pixel values of the input images. In all experiments we have no error states in spite of the number of reference patterns exceeds the classical estimations for traditional BAM and Hopfield networks.

Clique-Based Neural Associative Memories with Local Coding and Precoding.

Techniques from coding theory are able to improve the efficiency of neuroinspired and neural associative memories by forcing some construction and constraints on the network. In this letter, the approach is to embed coding techniques into neural associative memory in order to increase their performance in the presence of partial erasures. The motivation comes from recent work by Gripon, Berrou, and coauthors, which revisited Willshaw networks and presented a neural network with interacting neurons that partitioned into clusters. The model introduced stores patterns as small-size cliques that can be retrieved in spite of partial error. We focus on improving the success of retrieval by applying two techniques: doing a local coding in each cluster and then applying a precoding step. We use a slightly different decoding scheme, which is appropriate for partial erasures and converges faster. Although the ideas of local coding and precoding are not new, the way we apply them is different. Simulations show an increase in the pattern retrieval capacity for both techniques. Moreover, we use self-dual additive codes over field [Formula: see text], which have very interesting properties and a simple-graph representation.

Threshold Complex-Valued Neural Associative Memory

In this brief, threshold complex-valued neural associative memory is proposed for information retrieval. The introduction of threshold improves network performance by excluding rotated patterns from spurious memories. A design method for constructing different types of network is developed based on complex matrix decomposition, which is capable of designing nonthreshold, threshold, non-Hermitian, and Hermitian networks. Further, we illustrate the performance of the proposed method by reconstructing noisy 256 grayscale and true color images. The results show that constructed networks can work efficiently, threshold networks have better performance than nonthreshold ones and networks with small asymmetry in weight matrix function as well as Hermitian ones.

ENAMeL: A Language for Binary Correlation Matrix Memories

Despite their relative simplicity, correlation matrix memories (CMMs) are an active area of research, as they are able to be integrated into more complex architectures such as the Associative Rule Chaining Architecture (ARCA) “Austin et al. (International conference on artificial neural networks, pp 49–56, 2012)”. In this architecture, CMMs are used effectively in order to reduce the time complexity of a tree search from $$O(b^d)$$ to $$O(d)$$ —where $$b$$ is the branching factor and $$d$$ is the depth of the tree. This paper introduces the Extended Neural Associative Memory Language (ENAMeL)—a domain specific language developed to ease development of applications using CMMs. We discuss various considerations required while developing the language, and techniques used to reduce the memory requirements of CMM-based applications. Finally we show that the memory requirements of ARCA when using the ENAMeL interpreter compare favourably to our original results “Austin et al. (International conference on artificial neural networks, pp 49–56, 2012)” run in MATLAB.

Holographic memory-based Bayesian optimization algorithm (HM-BOA) in dynamic environments

This paper presents a new evolutionary dynamic optimization algorithm, holographic memory-based Bayesian optimization algorithm (HM-BOA), whose objective is to address the weaknesses of sequential memorybased dynamic optimization approaches. To this end, holographic associative neural memory is applied to one of the recent successful memory-based evolutionary methods, DBN-MBOA (memory-based BOA with dynamic Bayesian networks). Holographic memory is appropriate for encoding environmental changes since its stimulus and response data are represented by a vector of complex numbers such that the phase and the magnitude denote the information and its confidence level, respectively. In the learning process in HM-BOA, holographic memory is trained by probabilistic models at every environmental change. Its weight matrix contains abstract information obtained from previous changes and is used for constructing a new probabilistic model when the environment changes. The unique features of HM-BOA are: 1) the stored information can be generalized, and 2) a small amount of memory is required for storing the probabilistic models. Experimental results adduce grounds for its effectiveness especially in random environments.