Memory Augmented Neural Networks Research Articles

Application of the time-distributed layer in the controller of memory-augmented neural networks to classify brain activities into motor imagery and motor execution

Brain-Computer Interface (BCI) systems create a bridge between the human brain and the outside world, potentially rendering traditional methods of information transmission obsolete in the not-so-distant future. One of the key research areas in BCI is the classification of brain activity in electroencephalographic (EEG) data. On the other hand, new memory-augmented neural networks, such as the Neural Turing Machine (NTM) and the Differentiable Neural Computer (DNC), have demonstrated their impressive abilities in solving complex tasks. Therefore, it is useful to evaluate the capability of memory-augmented neural networks to enhance the classification of brain activity within EEG signals. Previous methods have suffered from low accuracy and generalizability in classifying brain activities; primarily due to a lack of proper classification of Motor Imagery/Execution brain activities, an inability to extract valuable information at different time steps in time series data, and a failure to learn from longer dependencies. This article introduces TDMANN (Time-Distributed Memory Augmented Neural Network), a framework that leverages the principles of NTM and DNC for the binary classification of brain activities in EEG signals. The controller component of the memory-augmented neural network is enhanced with a time-distributed approach, which significantly improves the performance of the network in binary classification tasks involving motor imagery/execution brain activities by extracting valuable information at each time step. The benchmark datasets used in this study are EEGmmidb BCI2000 (Imagery/Execution), BCI IV 2B, and BCI IV 2A, all containing motor imagery/execution brain activity data in EEG format. The results demonstrate that the classification accuracy achieved by the proposed DNC@TDMANN method exhibits a maximum improvement of 23.03% compared to baseline research works. The NTM@TDMANN method also shows a maximum improvement accuracy of 22.5%.

Decoupled and Explainable Associative Memory for Effective Knowledge Propagation.

Long-term memory often plays a pivotal role in human cognition through the analysis of contextual information. Machine learning researchers have attempted to emulate this process through the development of memory-augmented neural networks (MANNs) to leverage indirectly related but resourceful historical observations during learning and inference. The area of MANN, however, is still in its infancy and significant research effort is required to enable machines to achieve performance close to the human cognition process. This article presents an innovative MANN framework for the advanced incorporation of historical knowledge into a predictive framework. Within the key-value memory structure, we propose to decouple the key representations from the learned value memory embeddings to offer improved associations between the inputs and latent memory embeddings. We argue that the keys should be static, sparse, and unique representations of a particular observation to offer robust input to memory associations, while the value embeddings could be trainable, dense latent vectors such that they can better capture historical knowledge. Moreover, we introduce a novel memory update procedure that preserves the explainability of the historical knowledge extraction process, which would enable the human end-users to interpret the deep machine learning model decisions, fostering their trust. With extensive experiments conducted on three different datasets using audio, text, and image modalities, we demonstrate that our proposed innovations collectively allow this framework to outperform the current state-of-the-art methods by significant margins, irrespective of the modalities or the downstream tasks. The code is available at https://github.com/tha725/DE-KVMN/tree/main.

Generalized Key-Value Memory to Flexibly Adjust Redundancy in Memory-Augmented Networks.

Memory-augmented neural networks enhance a neural network with an external key-value (KV) memory whose complexity is typically dominated by the number of support vectors in the key memory. We propose a generalized KV memory that decouples its dimension from the number of support vectors by introducing a free parameter that can arbitrarily add or remove redundancy to the key memory representation. In effect, it provides an additional degree of freedom to flexibly control the tradeoff between robustness and the resources required to store and compute the generalized KV memory. This is particularly useful for realizing the key memory on in-memory computing hardware where it exploits nonideal, but extremely efficient nonvolatile memory devices for dense storage and computation. Experimental results show that adapting this parameter on demand effectively mitigates up to 44% nonidealities, at equal accuracy and number of devices, without any need for neural network retraining.

A2P-MANN: Adaptive Attention Inference Hops Pruned Memory-Augmented Neural Networks.

In this work, to limit the number of required attention inference hops in memory-augmented neural networks, we propose an online adaptive approach called [Formula: see text]-memory-augmented neural network (MANN). By exploiting a small neural network classifier, an adequate number of attention inference hops for the input query are determined. The technique results in the elimination of a large number of unnecessary computations in extracting the correct answer. In addition, to further lower computations in [Formula: see text]-MANN, we suggest pruning weights of the final fully connected (FC) layers. To this end, two pruning approaches, one with negligible accuracy loss and the other with controllable loss on the final accuracy, are developed. The efficacy of the technique is assessed by applying it to two different MANN structures and two question answering (QA) datasets. The analytical assessment reveals, for the two benchmarks, on average, 50% fewer computations compared to the corresponding baseline MANNs at the cost of less than 1% accuracy loss. In addition, when used along with the previously published zero-skipping technique, a computation count reduction of approximately 70% is achieved. Finally, when the proposed approach (without zero skipping) is implemented on the CPU and GPU platforms, on average, a runtime reduction of 43% is achieved.

MAN: Memory-augmented Attentive Networks for Deep Learning-based Knowledge Tracing

Knowledge Tracing (KT) is the task of modeling a learner’s knowledge state to predict future performance in e-learning systems based on past performance. Deep learning-based methods, such as recurrent neural networks, memory-augmented neural networks, and attention-based neural networks, have recently been used in KT. Such methods have demonstrated excellent performance in capturing the latent dependencies of a learner’s knowledge state on recent exercises. However, these methods have limitations when it comes to dealing with the so-called Skill Switching Phenomenon (SSP), i.e., when learners respond to exercises in an e-learning system, the latent skills in the exercises typically switch irregularly. SSP will deteriorate the performance of deep learning-based approaches for simulating the learner’s knowledge state during skill switching, particularly when the association between the switching skills and the previously learned skills is weak. To address this problem, we propose the Memory-augmented Attentive Network (MAN), which combines the advantages of memory-augmented neural networks and attention-based neural networks. Specifically, in MAN, memory-augmented neural networks are used to model learners’ longer term memory knowledge, while attention-based neural networks are used to model learners’ recent term knowledge. In addition, we design a context-aware attention mechanism that automatically weighs the tradeoff between these two types of knowledge. With extensive experiments on several e-learning datasets, we show that MAN effectively improve predictive accuracies of existing state-of-the-art DLKT methods.

Disentangling Features for Fashion Recommendation

Online stores have become fundamental for the fashion industry, revolving around recommendation systems to suggest appropriate items to customers. Such recommendations often suffer from a lack of diversity and propose items that are similar to previous purchases of a user. Recently, a novel kind of approach based on Memory Augmented Neural Networks (MANNs) has been proposed, aimed at recommending a variety of garments to create an outfit by complementing a given fashion item. In this article we address the task of compatible garment recommendation developing a MANN architecture by taking into account the co-occurrence of clothing attributes, such as shape and color, to compose an outfit. To this end we obtain disentangled representations of fashion items and store them in external memory modules, used to guide recommendations at inference time. We show that our disentangled representations are able to achieve significantly better performance compared to the state of the art and also provide interpretable latent spaces, giving a qualitative explanation of the recommendations.

Experimentally validated memristive memory augmented neural network with efficient hashing and similarity search

Lifelong on-device learning is a key challenge for machine intelligence, and this requires learning from few, often single, samples. Memory-augmented neural networks have been proposed to achieve the goal, but the memory module must be stored in off-chip memory, heavily limiting the practical use. In this work, we experimentally validated that all different structures in the memory-augmented neural network can be implemented in a fully integrated memristive crossbar platform with an accuracy that closely matches digital hardware. The successful demonstration is supported by implementing new functions in crossbars, including the crossbar-based content-addressable memory and locality sensitive hashing exploiting the intrinsic stochasticity of memristor devices. Simulations show that such an implementation can be efficiently scaled up for one-shot learning on more complex tasks. The successful demonstration paves the way for practical on-device lifelong learning and opens possibilities for novel attention-based algorithms that were not possible in conventional hardware.

Memory‐augmented neural networks based dynamic complex image segmentation in digital twins for self‐driving vehicle

With the continuous increase of the amount of information, people urgently need to identify the information in the image in more detail in order to obtain richer information from the image. This work explores the dynamic complex image segmentation of self-driving vehicle under Digital Twins (DTs) based on Memory-augmented Neural Networks (MANNs), so as to further improve the performance of self-driving in intelligent transportation. In view of the complexity of the environment and the dynamic changes of the scene in intelligent transportation, this work constructs a segmentation model for dynamic complex image of self-driving vehicle under DTs based on MANNs by optimizing the Deep Learning algorithm and further combining with the DTs technology, so as to recognize the information in the environment image during the self-driving. Finally, the performance of the constructed model is analyzed by experimenting with different image datasets (PASCALVOC 2012, NYUDv2, PASCAL CONTEXT, and real self-driving complex traffic image data). The results show that compared with other classical algorithms, the established MANN-based model has an accuracy of about 85.80%, the training time is shortened to 107.00 s, the test time is 0.70 s, and the speedup ratio is high. In addition, the average algorithm parameter of the given energy function α=0.06 reaches the maximum value. Therefore, it is found that the proposed model shows high accuracy and short training time, which can provide experimental reference for future image visual computing and intelligent information processing.

Bi-CLKT: Bi-Graph Contrastive Learning based Knowledge Tracing

The goal of Knowledge Tracing (KT) is to estimate how well students have mastered a concept based on their historical learning of related exercises. The benefit of knowledge tracing is that students’ learning plans can be better organised and adjusted, and interventions can be made when necessary. With the recent rise of deep learning, Deep Knowledge Tracing (DKT) has utilised Recurrent Neural Networks (RNNs) to accomplish this task with some success. Other works have attempted to introduce Graph Neural Networks (GNNs) and redefine the task accordingly to achieve significant improvements. However, these efforts suffer from at least one of the following drawbacks: (1) they pay too much attention to details of the nodes rather than to high-level semantic information; (2) they struggle to effectively establish spatial associations and complex structures of the nodes; and (3) they represent either concepts or exercises only, without integrating them. Inspired by recent advances in self-supervised learning, we propose a Bi-Graph Contrastive Learning based Knowledge Tracing (Bi-CLKT) to address these limitations. Specifically, we design a two-layer comparative learning scheme based on an “exercise-to-exercise” (E2E) relational subgraph. It involves node-level contrastive learning of subgraphs to obtain discriminative representations of exercises, and graph-level contrastive learning to obtain discriminative representations of concepts. Moreover, we designed a joint contrastive loss to obtain better representations and hence better prediction performance. Also, we explored two different variants, using RNN and memory-augmented neural networks as the prediction layer for comparison to obtain better representations of exercises and concepts respectively. Extensive experiments on four real-world datasets show that the proposed Bi-CLKT and its variants outperform other baseline models.

In-Memory Nearest Neighbor Search With Nanoelectromechanical Ternary Content-Addressable Memory

Nearest neighbor (NN) search is widely used in pattern classification and memory-augmented neural networks. To overcome the von Neumann bottleneck in conventional NN search architecture, in this study, nanoelectromechanical-switch-based ternary content-addressable memory (NEMTCAM) is introduced for the NN classifier. NEMTCAM can calculate the Hamming distance between the input vector and the stored vectors in a parallel search operation. The NEMTCAM operation was experimentally demonstrated. Furthermore, an analytical model for NN search accuracy, including cell-to-cell parasitic resistance, is presented. NEMTCAM can calculate up to 10 Hamming distances in 32-bit words owing to the high current ratio of the NEM memory switches.

SAPIENS: A 64-kb RRAM-Based Non-Volatile Associative Memory for One-Shot Learning and Inference at the Edge

Learning from a few examples (one/few-shot learning) on the fly is a key challenge for on-device machine intelligence. We present the first chip-level demonstration of one-shot learning with Stanford Associative memory for Programmable, Integrated Edge iNtelligence via life-long learning and Search (SAPIENS), a resistive random access memory (RRAM)-based non-volatile associative memory (AM) chip that serves as the backend for memory-augmented neural networks (MANNs). The 64-kb fully integrated RRAM-CMOS AM chip performs long-term feature embedding and retrieval, demonstrated on a 32-way one-shot learning task on the Omniglot dataset. Using only one example per class for 32 unseen classes during on-chip learning, SAPIENS achieves 79% measured inference accuracy on Omniglot, comparable to edge software model accuracy using five-level quantization (82%). It achieves an energy efficiency of 118 GOPS/W at 200 MHz for in-memory L1 distance computation and prediction. Multi-bank measurements on the same chip show that increasing the capacity from three banks (24 kb) to eight banks (64 kb) improves the chip accuracy from 73.5% to 79%, while minimizing the accuracy excursion due to bank-to-bank variability.

Reservoir stack machines

Memory-augmented neural networks equip a recurrent neural network with an explicit memory to support tasks that require information storage without interference over long times. A key motivation for such research is to perform classic computation tasks, such as parsing. However, memory-augmented neural networks are notoriously hard to train, requiring many backpropagation epochs and a lot of data. In this paper, we introduce the reservoir stack machine, a model which can provably recognize all deterministic context-free languages and circumvents the training problem by training only the output layer of a recurrent net and employing auxiliary information during training about the desired interaction with a stack. In our experiments, we validate the reservoir stack machine against deep and shallow networks from the literature on three benchmark tasks for Neural Turing machines and six deterministic context-free languages. Our results show that the reservoir stack machine achieves zero error, even on test sequences longer than the training data, requiring only a few seconds of training time and 100 training sequences.

Detecting and explaining unfairness in consumer contracts through memory networks

Recent work has demonstrated how data-driven AI methods can leverage consumer protection by supporting the automated analysis of legal documents. However, a shortcoming of data-driven approaches is poor explainability. We posit that in this domain useful explanations of classifier outcomes can be provided by resorting to legal rationales. We thus consider several configurations of memory-augmented neural networks where rationales are given a special role in the modeling of context knowledge. Our results show that rationales not only contribute to improve the classification accuracy, but are also able to offer meaningful, natural language explanations of otherwise opaque classifier outcomes.

Robust high-dimensional memory-augmented neural networks

Traditional neural networks require enormous amounts of data to build their complex mappings during a slow training procedure that hinders their abilities for relearning and adapting to new data. Memory-augmented neural networks enhance neural networks with an explicit memory to overcome these issues. Access to this explicit memory, however, occurs via soft read and write operations involving every individual memory entry, resulting in a bottleneck when implemented using the conventional von Neumann computer architecture. To overcome this bottleneck, we propose a robust architecture that employs a computational memory unit as the explicit memory performing analog in-memory computation on high-dimensional (HD) vectors, while closely matching 32-bit software-equivalent accuracy. This is achieved by a content-based attention mechanism that represents unrelated items in the computational memory with uncorrelated HD vectors, whose real-valued components can be readily approximated by binary, or bipolar components. Experimental results demonstrate the efficacy of our approach on few-shot image classification tasks on the Omniglot dataset using more than 256,000 phase-change memory devices. Our approach effectively merges the richness of deep neural network representations with HD computing that paves the way for robust vector-symbolic manipulations applicable in reasoning, fusion, and compression.

Memory-Augmented Neural Networks on FPGA for Real-Time and Energy-Efficient Question Answering

Memory-augmented neural networks (MANNs) were introduced to handle long-term dependent data efficiently. MANNs have shown promising results in question answering (QA) tasks that require holding contexts for answering a given question. As demands for QA on edge devices have increased, the utilization of MANNs in resource-constrained environments has become important. To achieve fast and energy-efficient inference of MANNs, we can exploit application-specific hardware accelerators on field-programmable gate arrays (FPGAs). Although several accelerators for conventional deep neural networks have been designed, it is difficult to efficiently utilize the accelerators with MANNs due to different requirements. In addition, characteristics of QA tasks should be considered for further improving the efficiency of inference on the accelerators. To address the aforementioned issues, we propose an inference accelerator of MANNs on FPGA. To fully utilize the proposed accelerator, we introduce fast inference methods considering the features of QA tasks. To evaluate our proposed approach, we implemented the proposed architecture on an FPGA and measured the execution time and energy consumption for the bAbI data set. According to our thorough experiments, the proposed methods improved speed and energy efficiency of the inference of MANNs up to about 25.6 and 28.4 times, respectively, compared with those of CPU.

Binarization Methods for Motor-Imagery Brain–Computer Interface Classification

Successful motor-imagery brain-computer interface (MI-BCI) algorithms either extract a large number of handcrafted features and train a classifier, or combine feature extraction and classification within deep convolutional neural networks (CNNs). Both approaches typically result in a set of real-valued weights, that pose challenges when targeting real-time execution on tightly resource-constrained devices. We propose methods for each of these approaches that allow transforming real-valued weights to binary numbers for efficient inference. Our first method, based on sparse bipolar random projection, projects a large number of real-valued Riemannian covariance features to a binary space, where a linear SVM classifier can be learned with binary weights too. By tuning the dimension of the binary embedding, we achieve almost the same accuracy in 4-class MI (≤1.27% lower) compared to models with float16 weights, yet delivering a more compact model with simpler operations to execute. Second, we propose to use memory-augmented neural networks (MANNs) for MI-BCI such that the augmented memory is binarized. Our method replaces the fully connected layer of CNNs with a binary augmented memory using bipolar random projection, or learned projection. Our experimental results on EEGNet, an already compact CNN for MI-BCI, show that it can be compressed by $1.28\times $ at iso-accuracy using the random projection. On the other hand, using the learned projection provides 3.89% higher accuracy but increases the memory size by $28.10\times $ .

Memory transformation networks for weakly supervised visual classification

The lack of labeled exemplars makes video classification based on supervised neural networks difficult and challenging. Utilizing external memory that contains task-related knowledge is a beneficial way to learn a category from a handful of samples; however, most existing memory-augmented neural networks still struggle to provide a satisfactory solution for multi-modal external data due to the high dimensionality and massive volume. In light of this, we propose a Memory Transformation Network (MTN) to convert external knowledge, by involving embedded and concentrated memories, so as to leverage it feasibly for video classification with weak supervision. Specifically, we employ a multi-modal deep autoencoder to project external visual and textual information onto a shared space to produce joint embedded memory, which can capture the correlation amongst different modalities to enhance the expressive ability. The curse of dimensionality issue can also be alleviated owing to the inherent dimension reduction ability of the autoencoder. Besides, an attention-based compression mechanism is employed to generate concentrated memory, which records useful information related to a specific task. In this way, the obtained concentrated memory is relatively lightweight to mitigate the time-consuming content-based addressing on large-volume memory. Our model outperforms the state-of-the-arts by 5.44% and 1.81% on average in two metrics over three real-world video datasets, demonstrating its effectiveness and superiority on visual classification with limited labeled exemplars.

FARM: A Flexible Accelerator for Recurrent and Memory Augmented Neural Networks

Recently, Memory Augmented Neural Networks (MANN)s, a class of Deep Neural Networks (DNN)s have become prominent owing to their ability to capture the long term dependencies effectively for several Natural Language Processing (NLP) tasks. These networks augment conventional DNNs by incorporating memory and attention mechanisms external to the network to capture relevant information. Several MANN architectures have shown particular benefits in NLP tasks by augmenting an underlying Recurrent Neural Network (RNN) with external memory using attention mechanisms. Unlike conventional DNNs whose computational time is dominated by MAC operations, MANNs have more diverse behavior. In addition to MACs, the attention mechanisms of MANNs also consist of operations such as similarity measure, sorting, weighted memory access, and pair-wise arithmetic. Due to this greater diversity in operations, MANNs are not trivially accelerated by the same techniques used by existing DNN accelerators. In this work, we present an end-to-end hardware accelerator architecture, FARM, for the inference of RNNs and several variants of MANNs, such as the Differential Neural Computer (DNC), Neural Turing Machine (NTM) and Meta-learning model. FARM achieves an average speedup of 30x-190x and 80x-100x over CPU and GPU implementations, respectively. To address remaining memory bottlenecks in FARM, we then propose the FARM-PIM architecture, which augments FARM with in-memory compute support for MAC and content-similarity operations in order to reduce data traversal costs. FARM-PIM offers an additional speedup of 1.5x compared to FARM. Additionally, we consider an efficiency-oriented version of the PIM implementation, FARM-PIM-LP, that trades a 20% performance reduction relative to FARM for a 4x average power consumption reduction.

Ferroelectric ternary content-addressable memory for one-shot learning

Deep neural networks are efficient at learning from large sets of labelled data, but struggle to adapt to previously unseen data. In pursuit of generalized artificial intelligence, one approach is to augment neural networks with an attentional memory so that they can draw on already learnt knowledge patterns and adapt to new but similar tasks. In current implementations of such memory augmented neural networks (MANNs), the content of a network’s memory is typically transferred from the memory to the compute unit (a central processing unit or graphics processing unit) to calculate similarity or distance norms. The processing unit hardware incurs substantial energy and latency penalties associated with transferring the data from the memory and updating the data at random memory addresses. Here, we show that ternary content-addressable memories (TCAMs) can be used as attentional memories, in which the distance between a query vector and each stored entry is computed within the memory itself, thus avoiding data transfer. Our compact and energy-efficient TCAM cell is based on two ferroelectric field-effect transistors. We evaluate the performance of our ferroelectric TCAM array prototype for one- and few-shot learning applications. When compared with a MANN where cosine distance calculations are performed on a graphics processing unit, the ferroelectric TCAM approach provides a 60-fold reduction in energy and 2,700-fold reduction in latency for a single memory search operation. A compact ternary content-addressable memory cell, which is based on two ferroelectric field-effect transistors, can provide memory augmented neural networks with improved energy and latency performance compared with traditional approaches based on graphics processing units.

Recurrent Highway Networks With Grouped Auxiliary Memory

Recurrent neural networks (RNNs) are challenging to train, let alone those with deep spatial structures. Architectures built upon highway connections such as Recurrent Highway Network (RHN) were developed to allow larger step-to-step transition depth, leading to more expressive models. However, problems that require capturing long-term dependencies still cannot be well addressed by these models. Moreover, the ability to keep long-term memories tends to diminish when the spatial depth increases, since deeper structure may accelerate gradient vanishing. In this paper, we address these issues by proposing a novel RNN architecture based on RHN, namely the Recurrent Highway Network with Grouped Auxiliary Memory (GAM-RHN). The proposed architecture interconnects the RHN with a set of auxiliary memory units specifically for storing long-term information via reading and writing operations, which is analogous to Memory Augmented Neural Networks (MANNs). Experimental results on artificial long time lag tasks show that GAM-RHNs can be trained efficiently while being deep in both time and space. We also evaluate the proposed architecture on a variety of tasks, including language modeling, sequential image classification, and financial market forecasting. The potential of our approach is demonstrated by achieving state-of-the-art results on these tasks.