Neural Turing Machine Research Articles

Recognition of words from brain-generated signals of speech-impaired people: Application of autoencoders as a neural Turing machine controller in deep neural networks

There is an essential requirement to support people with speech and communication disabilities. A brain–computer interface using electroencephalography (EEG) is applied to satisfy this requirement. A number of research studies to recognize brain signals using machine learning and deep neural networks (DNNs) have been performed to increase the brain signal detection rate, yet there are several defects and limitations in the techniques. Among them is the use in specific circumstances of machine learning. On the one hand, DNNs extract the features well and automatically. On the other hand, their use results in overfitting and vanishing problems. Consequently, in this research, a deep network is designed on the basis of an autoencoder neural Turing machine (DN-AE-NTM) to resolve the problems by the use of NTM external memory. In addition, the DN-AE-NTM copes with all kinds of signals with high detection rates. The data were collected by P300 EEG devices from several individuals under the same conditions. During the test, each individual was requested to skim images with one to six labels and focus on only one of the images. Not to focus on some images is analogous to producing unimportant information in the individual’s brain, which provides unfamiliar signals. Besides the main P300 EEG dataset, EEG recordings of individuals with alcoholism and control individuals and the EEGMMIDB, MNIST, and ORL datasets were implemented and tested. The proposed DN-AE-NTM method classifies data with an average detection rate of 97.5%, 95%, 98%, 99.4%, and 99.1%, respectively, in situations where the signals are noisy so that only 20% of the data are reliable and include useful information.

A Taxonomy for Neural Memory Networks.

An increasing number of neural memory networks have been developed, leading to the need for a systematic approach to analyze and compare their underlying memory structures. Thus, in this paper, we first create a framework for memory organization and then compare four popular dynamic models: vanilla recurrent neural network, long short-term memory, neural stack, and neural RAM. This analysis helps to open the dynamic neural networks' black box from the memory usage prospective. Accordingly, a taxonomy for these networks and their variants is proposed and proved using a unifying architecture. With the taxonomy, both network architectures and learning tasks are classified into four classes, and a one-to-one mapping is built between them to help practitioners select the appropriate architecture. To exemplify each task type, four synthetic tasks with different memory requirements are selected. Moreover, we use some signal processing applications and two natural language processing applications to evaluate the methodology in a realistic setting.

A classifier task based on Neural Turing Machine and particle swarm algorithm

Deep learning in artificial intelligence looks for a general-purpose computational machine to execute complex algorithms similar to humans’ brain. Neural Turing Machine (NTM) as a tool to realize deep learning approach brings together Turing machine that is a general-purpose machine equipped to a long-term memory, and a neural network as a controller. NTM applies simple controllers to execute several simple/complex tasks such as copy, sort, N-gram, etc.; however, complex tasks such as classifications are neglected, and there is no control over improving the weights of NTM, either. This paper presents a framework called PSONTM that improves the accuracy of NTM using the LSTM deep neural network as the controller, and implements a complex classification task along with available tasks. Particle Swarm Optimization (PSO) algorithm is also applied in order to control the weights. The classifier task is compared with the basic SVM, KNN, Naïve Bayesian, and Decision Tree classification methods on MNIST, ORL, letter recognition, and ionosphere datasets. The accuracy of the proposed classification tasks is 99.73%, 97.9%, 99.02%, and 97.1%, respectively. That means the NTM classification task improved Naïve Bayesian 43.57%, Decision Tree 15.6%, and KNN 19.22% on average. In addition, the presented framework improves the available NTM tasks as well.

Emergent neural turing machine and its visual navigation

Traditional Turing Machines (TMs) are symbolic whose hand-crafted representations are static and limited. Developmental Network 1 (DN-1) uses emergent representation to perform Turing Computation. But DN-1 lacks hierarchy in its internal representations, and is hard to handle the complex visual navigation tasks. In this paper, we improve DN-1 with several new mechanisms and presenta new emergent neural Turing Machine — Developmental Network 2 (DN-2). By neural, we mean that the control of the TM has neurons as basic computing elements. The major novelty of DN-2 over DN-1 is that the representational hierarchy inside DN-2 is emergent and fluid. DN-2 grows complex hierarchies by dynamically allowing initialization of neurons with different ranges of connection. We present a complex task — vision guided navigation in simulated and natural worlds using DN-2. A major function that has not been demonstrated before is that the hierarchy enables attention that disregards distracting features based on the navigation context. In simulated navigation experiments, DN-2 can perform with no errors, and in real-world navigation experiments, the error rate is only 0.78%. These experimental results showed that DN-2 successfully learned rules of navigation with image inputs.

Neuroevolution of a Modular Memory-Augmented Neural Network for Deep Memory Problems.

We present Modular Memory Units (MMUs), a new class of memory-augmented neural network. MMU builds on the gated neural architecture of Gated Recurrent Units (GRUs) and Long Short Term Memory (LSTMs), to incorporate an external memory block, similar to a Neural Turing Machine (NTM). MMU interacts with the memory block using independent read and write gates that serve to decouple the memory from the central feedforward operation. This allows for regimented memory access and update, giving our network the ability to choose when to read from memory, update it, or simply ignore it. This capacity to act in detachment allows the network to shield the memory from noise and other distractions, while simultaneously using it to effectively retain and propagate information over an extended period of time. We train MMU using both neuroevolution and gradient descent, and perform experiments on two deep memory benchmarks. Results demonstrate that MMU performs significantly faster and more accurately than traditional LSTM-based methods, and is robust to dramatic increases in the sequence depth of these memory benchmarks.

Multi-Channel Encoder for Neural Machine Translation

Attention-based Encoder-Decoder has the effective architecture for neural machine translation (NMT), which typically relies on recurrent neural networks (RNN) to build the blocks that will be lately called by attentive reader during the decoding process. This design of encoder yields relatively uniform composition on source sentence, despite the gating mechanism employed in encoding RNN. On the other hand, we often hope the decoder to take pieces of source sentence at varying levels suiting its own linguistic structure: for example, we may want to take the entity name in its raw form while taking an idiom as a perfectly composed unit. Motivated by this demand, we propose Multi-channel Encoder (MCE), which enhances encoding components with different levels of composition. More specifically, in addition to the hidden state of encoding RNN, MCE takes 1) the original word embedding for raw encoding with no composition, and 2) a particular design of external memory in Neural Turing Machine NTM) for more complex composition, while all three encoding strategies are properly blended during decoding. Empirical study on Chinese-English translation shows that our model can improve by 6.52 BLEU points upon a strong open source NMT system: DL4MT1. On the WMT14 English-French task, our single shallow system achieves BLEU=38.8, comparable with the state-of-the-art deep models.

Dynamic Neural Turing Machine with Continuous and Discrete Addressing Schemes.

We extend the neural Turing machine (NTM) model into a dynamic neural Turing machine (D-NTM) by introducing trainable address vectors. This addressing scheme maintains for each memory cell two separate vectors, content and address vectors. This allows the D-NTM to learn a wide variety of location-based addressing strategies, including both linear and nonlinear ones. We implement the D-NTM with both continuous and discrete read and write mechanisms. We investigate the mechanisms and effects of learning to read and write into a memory through experiments on Facebook bAbI tasks using both a feedforward and GRU controller. We provide extensive analysis of our model and compare different variations of neural Turing machines on this task. We show that our model outperforms long short-term memory and NTM variants. We provide further experimental results on the sequential [Formula: see text]MNIST, Stanford Natural Language Inference, associative recall, and copy tasks.

On the Duration, Addressability, and Capacity of Memory-Augmented Recurrent Neural Networks

Memory-augmented recurrent neural networks (M-RNNs) have demonstrated empirically that they are very attractive for many applications, but a good theoretical understanding of their behaviors is unclear yet. In this paper, three analytical indicators named duration, addressability, and capacity of general forms of the additional memory in M-RNNs are formalized. The analysis results of the interactions among these indicators reveal that it is hard for an M-RNN to simultaneously provide good performance on more than two out of three of indicators. Meanwhile, the duration, addressability, and capacity are applied to analyze and compare two M-RNNs: long short term memory and neural turing machine for different cases. The comparison results show that none of the models has better performance on one indicator than the other model all the time. Moreover, it is found that separating memory system into sub-memories can bring the increasing duration and addressability and the decreasing capacity for the whole memory system.