Memory-augmented Neural Network Research Articles

A2P-MANN: Adaptive Attention Inference Hops Pruned Memory-Augmented Neural Networks.

In this work, to limit the number of required attention inference hops in memory-augmented neural networks, we propose an online adaptive approach called [Formula: see text]-memory-augmented neural network (MANN). By exploiting a small neural network classifier, an adequate number of attention inference hops for the input query are determined. The technique results in the elimination of a large number of unnecessary computations in extracting the correct answer. In addition, to further lower computations in [Formula: see text]-MANN, we suggest pruning weights of the final fully connected (FC) layers. To this end, two pruning approaches, one with negligible accuracy loss and the other with controllable loss on the final accuracy, are developed. The efficacy of the technique is assessed by applying it to two different MANN structures and two question answering (QA) datasets. The analytical assessment reveals, for the two benchmarks, on average, 50% fewer computations compared to the corresponding baseline MANNs at the cost of less than 1% accuracy loss. In addition, when used along with the previously published zero-skipping technique, a computation count reduction of approximately 70% is achieved. Finally, when the proposed approach (without zero skipping) is implemented on the CPU and GPU platforms, on average, a runtime reduction of 43% is achieved.

Flash-based content addressable memory with L2 distance for memory-augmented neural network

Memory-augmented neural network (MANN) has received increasing attention as a promising approach to achieve lifelong on-device learning, of which implementation of the explicit memory is vital. Content addressable memory (CAM) has been designed to accelerate the explicit memory by harnessing the in-memory-computing capability. In this work, a CAM cell with quadratic code is proposed, and a 1Mb Flash-based multi-bit CAM chip capable of computing Euclidean (L2) distance is fabricated. Compared with ternary CAM, the latency and energy are significantly reduced by 5.3- and 46.6-fold, respectively, for the MANN on Omniglot dataset. Besides, the recognition accuracy has slight degradation (<1%) even after baking for 105 s at 200°C, demonstrating the robustness to environmental disturbance. Performance evaluation indicates a reduction of 471-fold in latency and 1267-fold in energy compared with GPU for search operation. The proposed robust and energy-efficient CAM provides a promising solution to implement lifelong on-device machine intelligence.

Memory Augmented Neural Network Adaptive Controllers: Performance and Stability

In this article, we propose a novel control architecture, inspired from neuroscience, for adaptive control of continuous-time systems. The proposed architecture, in the setting of standard neural network (NN)-based adaptive control, augments an <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">external working memory</i> to the NN. The controller, through a write operation, writes the hidden layer feature vector of the NN to the external working memory and can also update this information with the observed error in the output. Through a read operation, the controller retrieves information from the working memory to modify the final control signal. First, we consider a simpler estimation problem to theoretically study the effect of an external memory and prove that the estimation accuracy can be improved by incorporating memory. We, then, consider a model reference NN adaptive controller for linear systems with matched uncertainty to implement and illustrate our ideas. We prove that the resulting controller leads to a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">uniformly bounded</i> stable closed-loop system. Through extensive simulations and specific metrics, such as peak deviation and settling time, we show that memory augmentation improves learning significantly. Importantly, we also provide evidence for and insights on the mechanism by which this specific memory augmentation improves learning.

Structural plasticity‐based hydrogel optical Willshaw model for one‐shot on‐the‐fly edge learning

AbstractAutonomous one‐shot on‐the‐fly learning copes with the high privacy, small dataset, and in‐stream data at the edge. Implementing such learning on digital hardware suffers from the well‐known von‐Neumann and scaling bottlenecks. The optical neural networks featuring large parallelism, low latency, and high efficiency offer a promising solution. However, ex‐situ training of conventional optical networks, where optical path configuration and deep learning model optimization are separated, incurs hardware, energy and time overheads, and defeats the advantages in edge learning. Here, we introduced a bio‐inspired material‐algorithm co‐design to construct a hydrogel‐based optical Willshaw model (HOWM), manifesting Hebbian‐rule‐based structural plasticity for simultaneous optical path configuration and deep learning model optimization thanks to the underlying opto‐chemical reactions. We first employed the HOWM as an all optical in‐sensor AI processor for one‐shot pattern classification, association and denoising. We then leveraged HOWM to function as a ternary content addressable memory (TCAM) of an optical memory augmented neural network (MANN) for one‐shot learning the Omniglot dataset. The HOWM empowered one‐shot on‐the‐fly edge learning leads to 1000× boost of energy efficiency and 10× boost of speed, which paves the way for the next‐generation autonomous, efficient, and affordable smart edge systems.image

Experimentally validated memristive memory augmented neural network with efficient hashing and similarity search

Lifelong on-device learning is a key challenge for machine intelligence, and this requires learning from few, often single, samples. Memory-augmented neural networks have been proposed to achieve the goal, but the memory module must be stored in off-chip memory, heavily limiting the practical use. In this work, we experimentally validated that all different structures in the memory-augmented neural network can be implemented in a fully integrated memristive crossbar platform with an accuracy that closely matches digital hardware. The successful demonstration is supported by implementing new functions in crossbars, including the crossbar-based content-addressable memory and locality sensitive hashing exploiting the intrinsic stochasticity of memristor devices. Simulations show that such an implementation can be efficiently scaled up for one-shot learning on more complex tasks. The successful demonstration paves the way for practical on-device lifelong learning and opens possibilities for novel attention-based algorithms that were not possible in conventional hardware.

StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts

Inferring spatial relations in natural language is a crucial ability an intelligent system should possess. The bAbI dataset tries to capture tasks relevant to this domain (task 17 and 19). However, these tasks have several limitations. Most importantly, they are limited to fixed expressions, they are limited in the number of reasoning steps required to solve them, and they fail to test the robustness of models to input that contains irrelevant or redundant information. In this paper, we present a new Question-Answering dataset called StepGame for robust multi-step spatial reasoning in texts. Our experiments demonstrate that state-of-the-art models on the bAbI dataset struggle on the StepGame dataset. Moreover, we propose a Tensor-Product based Memory-Augmented Neural Network (TP-MANN) specialized for spatial reasoning tasks. Experimental results on both datasets show that our model outperforms all the baselines with superior generalization and robustness performance.

A neural network model of when to retrieve and encode episodic memories.

Recent human behavioral and neuroimaging results suggest that people are selective in when they encode and retrieve episodic memories. To explain these findings, we trained a memory-augmented neural network to use its episodic memory to support prediction of upcoming states in an environment where past situations sometimes reoccur. We found that the network learned to retrieve selectively as a function of several factors, including its uncertainty about the upcoming state. Additionally, we found that selectively encoding episodic memories at the end of an event (but not mid-event) led to better subsequent prediction performance. In all of these cases, the benefits of selective retrieval and encoding can be explained in terms of reducing the risk of retrieving irrelevant memories. Overall, these modeling results provide a resource-rational account of why episodic retrieval and encoding should be selective and lead to several testable predictions.

Distributed associative memory network with memory refreshing loss

Despite recent progress in memory augmented neural network (MANN) research, associative memory networks with a single external memory still show limited performance on complex relational reasoning tasks. Especially the content-based addressable memory networks often fail to encode input data into rich enough representation for relational reasoning and this limits the relation modeling performance of MANN for long temporal sequence data. To address these problems, here we introduce a novel Distributed Associative Memory architecture (DAM) with Memory Refreshing Loss (MRL) which enhances the relation reasoning performance of MANN. Inspired by how the human brain works, our framework encodes data with distributed representation across multiple memory blocks and repeatedly refreshes the contents for enhanced memorization similar to the rehearsal process of the brain. For this procedure, we replace a single external memory with a set of multiple smaller associative memory blocks and update these sub-memory blocks simultaneously and independently for the distributed representation of input data. Moreover, we propose MRL which assists a task’s target objective while learning relational information existing in data. MRL enables MANN to reinforce an association between input data and task objective by reproducing stochastically sampled input data from stored memory contents. With this procedure, MANN further enriches the stored representations with relational information. In experiments, we apply our approaches to Differential Neural Computer (DNC), which is one of the representative content-based addressing memory models and achieves the state-of-the-art performance on both memorization and relational reasoning tasks.

Vision-Guided MPC for Robotic Path Following Using Learned Memory-Augmented Model.

The control of the interaction between the robot and environment, following a predefined geometric surface path with high accuracy, is a fundamental problem for contact-rich tasks such as machining, polishing, or grinding. Flexible path-following control presents numerous applications in emerging industry fields such as disassembly and recycling, where the control system must adapt to a range of dissimilar object classes, where the properties of the environment are uncertain. We present an end-to-end framework for trajectory-independent robotic path following for contact-rich tasks in the presence of parametric uncertainties. We formulate a combination of model predictive control with image-based path planning and real-time visual feedback, based on a learned state-space dynamic model. For modeling the dynamics of the robot-environment system during contact, we introduce the application of the differentiable neural computer, a type of memory augmented neural network (MANN). Although MANNs have been as yet unexplored in a control context, we demonstrate a reduction in RMS error of 21.0% compared with an equivalent Long Short-Term Memory (LSTM) architecture. Our framework was validated in simulation, demonstrating the ability to generalize to materials previously unseen in the training dataset.

WITHDRAWN: An integrated approach for weather forecasting and disaster prediction using deep learning architecture based on memory Augmented Neural Network’s (MANN’s)

Abstract Weather forecasting is a real time challenge that has been proven to be worst among the disaster in world over the last decade. Prediction begins even more complicated due to the ever-changing weather conditions. Many attributes have been negotiated with weather forecasting data that consider related attributes as independent variables. It is the cause of such natural disasters floods and droughts that meet people across globe every year. The higher accuracy of predicting rainfall is importance for countries like India as their economy depending on agriculture. Because of the mighty nature, few Statistical and Mathematical techniques fail to provide good rainfall accuracy prediction. The imbalance data of rainfall makes Artificial Neural Network, Recurrent Neural Networks (RNN), MANN’s (LSTM, GRU, NTM) the best process. An effective climate analysis is required to understand the various factors that contribute to climate change. It is therefore necessary to identify the relationship among these qualities to better understanding of the weather data. The purpose of this paper to give non-professionals easy access to strategies as well methods used in the field of rain forecast and compare various results of various methods and algorithms used in research.

Shear loading detection of through bolts in bridge structures using a percussion‐based one‐dimensional memory‐augmented convolutional neural network

AbstractThe through bolt, which can be used as a shear connector, has attracted more attention since several accelerated bridge construction methods have been applied to renovate damaged bridges and construct new ones. Because current methods for shear loading detection of through bolts require constant deployment of sensors, the percussion‐based method may be a better alternative to improve the practicality and reduce costs. However, to process percussion‐induced sound signals, current percussion‐based methods all employ machine learning (ML) techniques that depend on manual extraction and classification of features. Attempting to solve this issue, we propose a one‐dimensional, memory‐augmented convolutional neural network (1D‐MACNN) inspired by the memory‐augmented neural network (MANN), which is the main computational novelty of this paper. Particularly, the proposed 1D‐MACNN has capacity to address new scenarios from unknown distributions, that is, the testing categories have not been seen during the training. By directly feeding the raw percussion‐induced sound signals into the 1D‐MACNN, the shear loading of through bolts can be detected. Compared to current ML‐based and deep learning‐based methods for one‐dimensional (1D) signals (e.g., 1D convolutional neural network and 1D convolutional neural network–long short‐term memory), the advantage of our proposed 1D‐MACNN is that it can achieve better performance. Specifically, the proposed 1D‐MACNN can achieve accuracy of 1, precision of 1, recall of 1, and F1‐score of 1. Moreover, the proposed 1D‐MACNN can effectively address the issue of new categories without retraining (in terms of two new categories: accuracy = .83; precision = .89; recall = .77; F1‐score = .83). Finally, the experimental results demonstrate the effectiveness of the 1D‐MACNN, which has great potential to detect shear loading of through bolts in bridge structures.

If structure can exclaim: a novel robotic-assisted percussion method for spatial bolt-ball joint looseness detection

In proportion to the immense construction of spatial structures is the emergence of catastrophes related to structural damages (e.g. loose connections), thus rendering personal injury and property loss. It is therefore essential to detect spatial bolt looseness. Current methods for detecting spatial bolt looseness mostly focus on contact-type measurement, which may not be practical in some cases. Thus, inspired by the sound-based human diagnostic approach, we develop a novel percussion method using the Mel-frequency cepstral coefficient and the memory-augmented neural network in this article. In comparison with current investigations, the main contribution of this article is the detection of multi-bolt looseness for the first time with higher accuracy than prior methods. In particular, in terms of new data obtained via similar joints, the memory-augmented neural network can help avoid inefficient relearn and assimilate new data to provide accurate prediction with only a few data samples, which effectively improves the robustness of detection. Furthermore, percussion was implemented with a robotic arm instead of manual operation, which preliminarily explores the potential of implementing automation applications in real industries. Finally, experimental results demonstrate the effectiveness of the proposed method, which can guide future development of cyber-physics systems for structural health detection.

Skeleton-Based Dynamic Hand Gesture Recognition Using an Enhanced Network with One-Shot Learning

Dynamic hand gesture recognition based on one-shot learning requires full assimilation of the motion features from a few annotated data. However, how to effectively extract the spatio-temporal features of the hand gestures remains a challenging issue. This paper proposes a skeleton-based dynamic hand gesture recognition using an enhanced network (GREN) based on one-shot learning by improving the memory-augmented neural network, which can rapidly assimilate the motion features of dynamic hand gestures. Besides, the network effectively combines and stores the shared features between dissimilar classes, which lowers the prediction error caused by the unnecessary hyper-parameters updating, and improves the recognition accuracy with the increase of categories. In this paper, the public dynamic hand gesture database (DHGD) is used for the experimental comparison of the state-of-the-art performance of the GREN network, and although only 30% of the dataset was used for training, the accuracy of skeleton-based dynamic hand gesture recognition reached 82.29% based on one-shot learning. Experiments with the Microsoft Research Asia (MSRA) hand gesture dataset verified the robustness of the GREN network. The experimental results demonstrate that the GREN network is feasible for skeleton-based dynamic hand gesture recognition based on one-shot learning.

Critical infrastructure protection based on memory-augmented meta-learning framework

Critical infrastructures are related to systems which are essential for sustaining the important functions of a society. Their potential failures can cause serious problems not only to the population and economy but also to national security as well. The importance of these infrastructures calls for measures toward their security and protection. The aim was the reduction of risk related to natural disasters, terrorist acts, and cyberthreats. Traditional security systems, even those that employ intelligent algorithms, fail to prevent advanced zero-day attacks as they require constant training. This research proposes a novel meta-learning architecture that considers the neural turing machines as the approach upon which the model is founded. The introduced model allows for the memorization of useful data from past processes, by integrating external storage memory. Moreover, it facilitates the rapid integration of new information without the need for retraining. In particular, the proposed novel architecture is called memory-augmented neural network (M-ANN) whose core is a sophisticated, very fast, and highly efficient extreme learning machine. The M-ANN is assisted by a series of original modifications, related to fine-tuning of training, to memory retrieval mechanisms, to addressing techniques, and to ways of attention-weight allocation to memory vectors. The efficiency of the proposed system has been successfully tested using an extremely complex scenario for the protection of critical infrastructures. According to the testing scenario, memory could quickly encode and record information about new types of attacks, while any stored representation from previous experience was easily and consistently accessible, to maximize the detection efficiency.

MANNWARE: A Malware Classification Approach with a Few Samples Using a Memory Augmented Neural Network

The ability to stop malware as soon as they start spreading will always play an important role in defending computer systems. It must be a huge benefit for organizations as well as society if intelligent defense systems could themselves detect and prevent new types of malware as soon as they reveal only a tiny amount of samples. An approach introduced in this paper takes advantage of One-shot/Few-shot learning algorithms to solve the malware classification problems using a Memory Augmented Neural Network in combination with the Natural Language Processing techniques such as word2vec, n-gram. We embed the malware’s API calls, which are very valuable sources of information for identifying malware’s behaviors, in the different feature spaces, and then feed them to the one-shot/few-shot learning models. Evaluating the model on the two datasets (FFRI 2017 and APIMDS) shows that the models with different parameters could yield high accuracy on malware classification with only a few samples. For example, on the APIMDS dataset, it was able to guess 78.85% correctly after seeing only nine malware samples and 89.59% after fine-tuning with a few other samples. The results confirmed very good accuracies compared to the other traditional methods, and point to a new area of malware research.

Knowledge Tracking Model Based on Learning Process

Knowledge tracking model has been a research hotspot in the field of educational data mining for a long time. Knowledge tracking can automatically discover students’ weak knowledge points, which helps to improve students’ self-motivation in learning and realize personalized guidance. The existing KT model has some shortcomings, such as the limitation of the calculation of knowledge growth, and the imperfect forgetting mechanism of the model. To this end, we proposed a new knowledge tracking model based on learning process (LPKT), LPKT applies the idea of Memory Augmented Neural Net-work(MANN).When we model the learning process of students, two additional important factors are considered. One is to consider the current state of knowledge of the students when updating the dynamic matrix of the neural network, and the other is to improve the forgetting mechanism of the model. In this paper we verified the effectiveness and superiority of LPKT through comparative experiments, and proved that the model can improve the effect of knowledge tracking and make the process of deep knowledge tracking easier to understand.

Concept learning through deep reinforcement learning with memory-augmented neural networks

Deep neural networks have shown superior performance in many regimes to remember familiar patterns with large amounts of data. However, the standard supervised deep learning paradigm is still limited when facing the need to learn new concepts efficiently from scarce data. In this paper, we present a memory-augmented neural network which is motivated by the process of human concept learning. The training procedure, imitating the concept formation course of human, learns how to distinguish samples from different classes and aggregate samples of the same kind. In order to better utilize the advantages originated from the human behavior, we propose a sequential process, during which the network should decide how to remember each sample at every step. In this sequential process, a stable and interactive memory serves as an important module. We validate our model in some typical one-shot learning tasks and also an exploratory outlier detection problem. In all the experiments, our model gets highly competitive to reach or outperform those strong baselines.

Neuroevolution of a Modular Memory-Augmented Neural Network for Deep Memory Problems.

We present Modular Memory Units (MMUs), a new class of memory-augmented neural network. MMU builds on the gated neural architecture of Gated Recurrent Units (GRUs) and Long Short Term Memory (LSTMs), to incorporate an external memory block, similar to a Neural Turing Machine (NTM). MMU interacts with the memory block using independent read and write gates that serve to decouple the memory from the central feedforward operation. This allows for regimented memory access and update, giving our network the ability to choose when to read from memory, update it, or simply ignore it. This capacity to act in detachment allows the network to shield the memory from noise and other distractions, while simultaneously using it to effectively retain and propagate information over an extended period of time. We train MMU using both neuroevolution and gradient descent, and perform experiments on two deep memory benchmarks. Results demonstrate that MMU performs significantly faster and more accurately than traditional LSTM-based methods, and is robust to dramatic increases in the sequence depth of these memory benchmarks.

Labeled Memory Networks for Online Model Adaptation

Augmenting a neural network with memory that can grow without growing the number of trained parameters is a recent powerful concept with many exciting applications. In this paper, we establish their potential in online adapting a batch trained neural network to domain-relevant labeled data at deployment time. We present the design of Labeled Memory Network (LMN), a new memory augmented neural network (MANN) for fast online model adaptation. We highlight three key features of LMNs. First, LMNs treat memory as a second boosted stage following the trained network thereby allowing the memory and network to play complementary roles. Unlike all existing MANNs that write to memory at every cycle, LMNs provide better memory utilization by writing only labeled data with non-zero loss. Second, LMNs organize the memory with the discrete class label as the primary key unlike existing MANNs where key is a real vector derived from the input. This simple, yet surprisingly unexplored alternative organization, safeguards against catastrophic forgetting of rare labels that current LRU based MANNs are subject to. Finally, LMNs model the evolving expertise of memory and network using a RNN, to determine online their respective weights we evaluate online model adaptation strategies on five sequence prediction tasks, an image classification task, and two language modeling tasks. We show that LMNs are better than other MANNs designed for meta-learning. We also found them to be more accurate and faster than state-of-the-art methods of retuning model parameters for adapting to domain-specific labeled data.