Neural Turing Machine Research Articles

Application of the time-distributed layer in the controller of memory-augmented neural networks to classify brain activities into motor imagery and motor execution

Brain-Computer Interface (BCI) systems create a bridge between the human brain and the outside world, potentially rendering traditional methods of information transmission obsolete in the not-so-distant future. One of the key research areas in BCI is the classification of brain activity in electroencephalographic (EEG) data. On the other hand, new memory-augmented neural networks, such as the Neural Turing Machine (NTM) and the Differentiable Neural Computer (DNC), have demonstrated their impressive abilities in solving complex tasks. Therefore, it is useful to evaluate the capability of memory-augmented neural networks to enhance the classification of brain activity within EEG signals. Previous methods have suffered from low accuracy and generalizability in classifying brain activities; primarily due to a lack of proper classification of Motor Imagery/Execution brain activities, an inability to extract valuable information at different time steps in time series data, and a failure to learn from longer dependencies. This article introduces TDMANN (Time-Distributed Memory Augmented Neural Network), a framework that leverages the principles of NTM and DNC for the binary classification of brain activities in EEG signals. The controller component of the memory-augmented neural network is enhanced with a time-distributed approach, which significantly improves the performance of the network in binary classification tasks involving motor imagery/execution brain activities by extracting valuable information at each time step. The benchmark datasets used in this study are EEGmmidb BCI2000 (Imagery/Execution), BCI IV 2B, and BCI IV 2A, all containing motor imagery/execution brain activity data in EEG format. The results demonstrate that the classification accuracy achieved by the proposed DNC@TDMANN method exhibits a maximum improvement of 23.03% compared to baseline research works. The NTM@TDMANN method also shows a maximum improvement accuracy of 22.5%.

Differentiable-Decision-Tree-Based Neural Turing Machine Model Integrated Into FDTD for Implementing EM Problems

As known to all, problems consume more and more computational resources, for example, memory and time, with the gradual enhancements of both the size of the physical domain and complexity in numerical methods. Computational efficiencies have to be obviously influenced, especially for the finite-difference time-domain (FDTD) solver due to its inflexibility in discretization principle. To further overcome this problem, the neural turing machine (NTM) model, based on the differentiable decision tree (DDT), is chosen and incorporated to improve the efficiencies during the FDTD solver in our work. The DDT-based NTM model has the remarkable advantages of both trees and neural networks, which can better explain for the numerical data. Meanwhile, it has full differentiability through neural networks and can be trained by the powerful deep-learning processes. The proposed DDT-based NTM model could not only enhance both accuracy and efficiency but also successfully integrate into the FDTD solver to implement the microstrip transmission line.

DAerosol-NTM: applying deep learning and neural Turing machine in aerosol prediction

DAerosol-NTM: applying deep learning and neural Turing machine in aerosol prediction

Air quality particulate-pollution prediction applying GAN network and the Neural Turing Machine

Urban areas in many countries face significant concerns over the presence of aerosol particles and their effects on human health. These particles, which range in size from 1 nanometer to 100 micrometers, can easily penetrate organic matter and transport toxic gas compounds and mineral substances such as carbon monoxide, ozone, nitrogen dioxide, and sulfur dioxide. High concentrations of airborne particles pose serious challenges to health, the economy, the environment, and society, and it is crucial to investigate ways to improve the quality of life for people. The main objective of this study is to develop a framework that accurately predicts aerosol and air quality index (AQI) values in advance by estimating missing data and predicting future data. To achieve this, we propose the DAerosol.GAN.NTM framework, which combines a neural Turing machine with a generative adversarial network (GAN) to address the limitations of previous studies. Our framework outperforms previous methods, including DAerosol.NTM (without GAN) and four baseline studies using Multilayer Perceptron (MLP), Deep Neural Networks (DNN), Long-short Term Memory (LSTM), and Deep LSTM (DLSTM) models, in terms of accuracy, precision, root mean square error (RMSE), mean absolute percentage error (MAPE), and the prediction of aerosol pollution surge hours in advance according to the Time Interval Before and After the Aerosol Event (TIBAAE) criterion.

Pan-Peptide Meta Learning for T-cell receptor–antigen binding recognition

The identification of the mechanisms by which T-cell receptors (TCRs) interact with human antigens provides a crucial opportunity to develop new vaccines, diagnostics and immunotherapies. However, the accurate prediction and recognition of TCR–antigen pairing represents a substantial computational challenge in immunology. Existing tools only learn the binding patterns of antigens from many known TCR binding repertoires and fail to recognize antigens that have never been presented to the immune system or for which only a few TCR binding repertoires are known. However, the binding specificity for neoantigens or exogenous peptides is crucial for immune studies and immunotherapy. Therefore, we developed Pan-Peptide Meta Learning (PanPep), a general and robust framework to recognize TCR–antigen binding, by combining the concepts of meta-learning and the neural Turing machine. The neural Turing machine adds external memory to avoid forgetting previously learned tasks, which is used here to accurately predict TCR binding specificity with any peptide, particularly unseen ones. We applied PanPep to various challenging clinical tasks, including (1) qualitatively measuring the clonal expansion of T cells; (2) efficiently sorting responsive T cells in tumour neoantigen therapy; and (3) accurately identifying immune-responsive TCRs in a large cohort from a COVID-19 study. Our comprehensive tests show that PanPep outperforms existing tools. PanPep also offers interpretability, revealing the nature of peptide and TCR interactions in 3D crystal structures. We believe PanPep can be a useful tool to decipher TCR–antigen interactions and that it has broad clinical applications. Machine learning methods can predict and recognize binding patterns between T-cell receptors and human antigens, but they struggle with antigens for which no or little data exist regarding interactions with the immune system. A new method called PanPep based on meta-learning can learn quickly on new binding prediction tasks and accurately predicts pairing between T-cell receptors and new antigens.

Real-time pricing method for VPP demand response based on PER-DDPG algorithm

Through advanced information communication and management system, virtual power plant (VPP) can realize the aggregation and coordination optimization of distributed energy, energy storage system, controllable load and other distributed energy resources. However, when making real-time price decisions according to users' demand response (DR) characteristics, the optimization effect of VPP is still limited by the evaluation accuracy of users’ DR potential and the computational burden of continuous decisions. By combining gate recurrent unit (GRU) and attention mechanism (AM), Neural Turing Machine (NTM) can extract users' response features in different environments and improve the accuracy of evaluating DR potential. Subsequently, based on the evaluation results, a deep deterministic policy gradient (DDPG) algorithm relying on prioritized experience replay (PER) is used to formulate a real-time electricity price plan. Ultimately, VPP achieves multi-objective optimization through DR management, which helps to increase the consumption amount of renewable energy resources, smooth its power fluctuation, and reduce users' electricity cost. Case study results show that the proposed method can improve the accuracy of the DR potential evaluation, reduce the response deviation to about 3%, and enhance the real-time decision calculation efficiency by 17%, which helps to optimize the smooth consumption of renewable energy.

Projection of early warning identification of hazardous sources of gas explosion accidents in coal mines based on NTM deep learning network

Abstract Among all kinds of coal production disasters, the consequences of gas disaster are the most serious. As the existing coal mine gas explosion disaster pre-control management theory and method system is not satisfactory, the neural Turing machine (NTM) deep learning network algorithm is used to calculate and analyse the risk source early warning identification of coal mine gas explosion accidents. Institute with data sets of gas gas accident knowledge base matter each event to cause an (basic or intermediate events) as an example, through the study of the depth of NTM network algorithm calculation analysis shows that self-rescuer failure, personnel peccancy operation, such as downhole safety management does not reach the designated position is easy to cause important hazard of gas explosion accident, the probability to cause an 0.567. Based on the constructed NTM deep learning network algorithm, the risk factors and their weights in the early warning identification of gas explosion accidents are calculated and analysed. Through calculation analysis, it can be seen that the highest weight of risk factors is gas concentration, with a weight of 96. In the early warning identification of hazard sources, the hazard factor next to gas concentration is mine combustibles, with a weight of 75.

A novel frequency-dependent hysteresis model based on improved neural Turing machine

A novel frequency-dependent hysteresis model based on improved neural Turing machine

Compound short- and long-term memory for memory augmented neural networks

Adding memory to artificial intelligence systems in an effective way has been addressed by researchers for many years. Recurrent neural networks and long short-term memories (LSTMs), among other neural network systems, have some inherent memory capabilities. Recently, in memory augmented neural networks, such as neural Turing machine (NTM) and its variants, a separate memory module is implemented, which can be accessed via read and write heads. Despite its capabilities in simple algorithmic tasks, such as copying and repeat copying, neural Turing machines fail when doing complex tasks with long-term dependencies due to their limited memory capacity. In this paper, we propose a new memory module in which data storing and access mechanisms are based on a graph-based neural structure rather than a matrix model. This is inspired by the human memory system, in which memories are stored via synapses (connections between neurons) and are recalled through a path passing from different neuronal networks. Differentiable mechanisms are designed for this graph-based neural memory so that it can be trained via backpropagation. The proposed structure is used to solve tasks with long-time dependencies and operations on the input sequences as well as the bAbI question answering dataset. The analysis shows that the proposed memory system can help solve these tasks better than NTM and LSTM in terms of the convergence speed and the final error

Testing technology for tensile properties of metal materials based on deep learning model.

The properties of metallic materials have been extensively studied, and nowadays the tensile properties testing techniques of metallic materials still have not found a suitable research method. In this paper, the neural Turing machine model is first applied to explore the tensile properties of metallic materials and its usability is demonstrated. Then the neural Turing machine model was improved. The model is then improved so that the required results can be obtained faster and more explicitly. Based on the improved Neural Turing Machine model in the exploration of tensile properties of metal materials, it was found that both H-NTM and AH-NTM have less training time than NTM. A-NTM takes more training time than AH-NTM. The improvement reduces the training time of the model. In replication, addition, and multiplication, the training time is reduced by 6.0, 8.8, and 7.3%, respectively. When the indentation interval is 0.5–0.7 mm, the error of the initial indentation data is large. The error of the tensile properties of the material obtained after removing the data at this time is significantly reduced. When the indentation interval is 0.8–1.5 mm, the stress is closer to the real value of tensile test yield strength 219.9 Mpa and tensile test tensile strength 258.8 Mpa. this paper will improve the neural Turing machine model in the exploration of metal material tensile properties testing technology has some application value.

Neural Turing Machines for the Remaining Useful Life estimation problem

Accurately estimating the amount of productive time remaining to a machine before it suffers a fault condition is a fundamental problem, which is commonly found in several contexts (e.g. mechanical systems) and can have great industrial, societal, and safety-related consequences. Recently, deep learning showed promising results and significant improvements towards a solution to the Remaining Useful Life estimation problem. In this paper, the usage of a sequence model called Neural Turing Machine (NTM), which can be seen as a “computer” that uses the available data to learn how to interact with an external memory, is thoroughly explored. In particular, even by using a single NTM as the key feature extraction component, more accurate solutions can be obtained when compared to widely used Long Short-Term Memory-based solutions. Moreover, such an improvement can be obtained while using fewer learnable parameters. The proposed approach is validated using sensor data of aircraft turbofan engines and particle filtration systems, obtaining competitive results to state-of-the-art techniques. Furthermore, the source code is released at https://github.com/aranciokov/NTM-For-RULEstimation to provide a strong baseline for the community, to support reproducibility and faster advancement in this field.

Memory augmented recurrent neural networks for de-novo drug design.

A recurrent neural network (RNN) is a machine learning model that learns the relationship between elements of an input series, in addition to inferring a relationship between the data input to the model and target output. Memory augmentation allows the RNN to learn the interrelationships between elements of the input over a protracted length of the input series. Inspired by the success of stack augmented RNN (StackRNN) to generate strings for various applications, we present two memory augmented RNN-based architectures: the Neural Turing Machine (NTM) and the Differentiable Neural Computer (DNC) for the de-novo generation of small molecules. We trained a character-level convolutional neural network (CNN) to predict the properties of a generated string and compute a reward or loss in a deep reinforcement learning setup to bias the Generator to produce molecules with the desired property. Further, we compare the performance of these architectures to gain insight to their relative merits in terms of the validity and novelty of the generated molecules and the degree of property bias towards the computational generation of de-novo drugs. We also compare the performance of these architectures with simpler recurrent neural networks (Vanilla RNN, LSTM, and GRU) without an external memory component to explore the impact of augmented memory in the task of de-novo generation of small molecules.

Adapting Neural Turing Machines for linguistic assessments aggregation in neural-symbolic decision support systems

Introduction: The construction of integrated neurosymbolic systems is an urgent and challenging task. Building neurosymbolic decision support systems requires new approaches to represent knowledge about a problem situation and to express symbolic reasoning at the subsymbolic level. Purpose: Development of neural network architectures and methods for effective distributed knowledge representation and subsymbolic reasoning in decision support systems in terms of algorithms for aggregation of fuzzy expert evaluations to select alternative solutions. Methods: Representation of fuzzy and uncertain estimators in a distributed form using tensor representations; construction of a trainable neural network architecture for subsymbolic aggregation of linguistic estimators. Results: The study proposes two new methods of representation of linguistic assessments in a distributed form. The first approach is based on the possibility of converting an arbitrary linguistic assessment into a numerical representation and consists in converting this numerical representation into a distributed one by converting the number itself into a bit string and further forming a matrix storing the distributed representation of the whole expression for aggregating the assessments. The second approach to translating linguistic assessments to a distributed representation is based on representing the linguistic assessment as a tree and coding this tree using the method of tensor representations, thus avoiding the step of translating the linguistic assessment into a numerical form and ensuring the transition between symbolic and subsymbolic representations of linguistic assessments without any loss of information. The structural elements of the linguistic assessment are treated as fillers with their respective positional roles. A new subsymbolic method of aggregation of linguistic assessments is proposed, which consists in creating a trainable neural network module in the form of a Neural Turing Machine. Practical relevance: The results of the study demonstrate how a symbolic algorithm for aggregation of linguistic evaluations can be implemented by connectionist (or subsymbolic) mechanisms, which is an essential requirement for building distributed neurosymbolic decision support systems.

Automatic Student Engagement in Online Learning Environment Based on Neural Turing Machine

With the continuous and rapid growth of online courses, online learners’ engagement recognition has become a novel research topic in the field of computer vision and pattern recognition. While a few attempts to automatic engagement recognition has been studied in the literature, learning a robust engagement measure is still a challenging task. To address it, we propose a new automatic engagement recognition method based on Neural Turing Machine in this paper. In particular, we firstly extract student’s eye gaze features, facial action unit features, head pose features, and body pose features respectively, then combine these multi modal features into the final feature of our recognition task. Moreover, we propose the engagement recognition framework based on the idea of Neural Turing Machine to learn the weight of each short video feature. In consequence, the feature fused by different weights will be applied to identify the students’ engagement in learning online courses. Empirically, we show improved performance over state of the art methods to automatic engagement recognition on DAiSEE dataset.

Language Model Using Neural Turing Machine Based on Localized Content-Based Addressing

The performance of a long short-term memory (LSTM) recurrent neural network (RNN)-based language model has been improved on language model benchmarks. Although a recurrent layer has been widely used, previous studies showed that an LSTM RNN-based language model (LM) cannot overcome the limitation of the context length. To train LMs on longer sequences, attention mechanism-based models have recently been used. In this paper, we propose a LM using a neural Turing machine (NTM) architecture based on localized content-based addressing (LCA). The NTM architecture is one of the attention-based model. However, the NTM encounters a problem with content-based addressing because all memory addresses need to be accessed for calculating cosine similarities. To address this problem, we propose an LCA method. The LCA method searches for the maximum of all cosine similarities generated from all memory addresses. Next, a specific memory area including the selected memory address is normalized with the softmax function. The LCA method is applied to pre-trained NTM-based LM during the test stage. The proposed architecture is evaluated on Penn Treebank and enwik8 LM tasks. The experimental results indicate that the proposed approach outperforms the previous NTM architecture.

A Review on Neural Turing Machine (NTM)

One of the significant objectives of artificial intelligence is to design learning algorithms that are executed on general-purpose computational machines inspired by the human brain. Neural Turing Machine (NTM) is a step towards realizing such a computational machine. In the literature, a variety of approaches have been presented for the NTM; however, there is no existing comprehensive survey and taxonomy for NTM methods. This article presents an overview of taxonomies characterizing the critical concepts of the NTM through a comprehensive survey on the related research activities. This in-depth analysis of taxonomies can provide researchers, designers, and application developers with a clear guideline to compare NTM methods. The taxonomy of machine learning, neural networks, and the Turing machine is introduced. The NTM is also inspected in terms of concepts, structure, implemented tasks, and related works. The article further presents research discussions and future challenges in this area.

A Dual-Stream architecture based on Neural Turing Machine and Attention for the Remaining Useful Life Estimation problem

Estimating in a reliable way the Remaining Useful Life (RUL) of a mechanical component is a fundamental task in the field of Prognostics and Health Management (PHM). In recent years a greater availability of high quality sensors and easiness of data gathering gave rise to data-driven models based on deep learning for this task, which has recently seen the introduction of "dual-stream" architectures. In this paper we propose a dual-stream architecture to address the RUL estimation problem through the exploitation of a Neural Turing Machine (NTM) and a Multi-Head Attention (MHA) mechanism. The NTM is a content-based memory addressing system which gives each of the streams the ability to access to and interact with the memory and acts as a fusion technique. The MHA is an attention mechanism added as a mean for our architecture to identify the existing relations between different sensor data in order to reveal hidden patterns among them. To evaluate the performance of our model, we considered the C-MAPSS dataset, a benchmark dataset published by NASA consisting of several time series related to the life of turbofan engines. We show that our approach achieves the best prediction score (which measures the safety of the predictions) in the available literature on two of the C-MAPSS subdatasets.

FARM: A Flexible Accelerator for Recurrent and Memory Augmented Neural Networks

Recently, Memory Augmented Neural Networks (MANN)s, a class of Deep Neural Networks (DNN)s have become prominent owing to their ability to capture the long term dependencies effectively for several Natural Language Processing (NLP) tasks. These networks augment conventional DNNs by incorporating memory and attention mechanisms external to the network to capture relevant information. Several MANN architectures have shown particular benefits in NLP tasks by augmenting an underlying Recurrent Neural Network (RNN) with external memory using attention mechanisms. Unlike conventional DNNs whose computational time is dominated by MAC operations, MANNs have more diverse behavior. In addition to MACs, the attention mechanisms of MANNs also consist of operations such as similarity measure, sorting, weighted memory access, and pair-wise arithmetic. Due to this greater diversity in operations, MANNs are not trivially accelerated by the same techniques used by existing DNN accelerators. In this work, we present an end-to-end hardware accelerator architecture, FARM, for the inference of RNNs and several variants of MANNs, such as the Differential Neural Computer (DNC), Neural Turing Machine (NTM) and Meta-learning model. FARM achieves an average speedup of 30x-190x and 80x-100x over CPU and GPU implementations, respectively. To address remaining memory bottlenecks in FARM, we then propose the FARM-PIM architecture, which augments FARM with in-memory compute support for MAC and content-similarity operations in order to reduce data traversal costs. FARM-PIM offers an additional speedup of 1.5x compared to FARM. Additionally, we consider an efficiency-oriented version of the PIM implementation, FARM-PIM-LP, that trades a 20% performance reduction relative to FARM for a 4x average power consumption reduction.

EDNC: Evolving Differentiable Neural Computers

One recent study in the field of deep learning is extending Artificial Neural Networks (ANNs) by coupling them to external memory resources. Neural Turing Machine (NTM) and Differentiable Neural Computer (DNC) are two counterparts in this field. Research activities fall into two categories of either interacting with memory or choosing controller components. The first approach uses a try and error fashion to provide the controller’s structure that would be a problem when there is no prior knowledge on the controller’s structure for particular tasks. The second approach includes methods for choosing the controller’s structure and weight optimization. NeuroEvolution falls in the second category that automatically and without prior knowledge obtains an appropriate network structure. This research presents Evolutionary Differentiable Neural Computer (EDNC), which uses a novel NeuroEvolution algorithm that is introduced in two types of nested object-oriented encoding called Adaptive Layer NeuroEvolution (ALNE) and Matrix-based one called M_ALNE. These NeuroEvolution algorithms use the following sub-algorithms: Self-Adaptivity in the Initial Population (SAIP), Self-Adaptivity in Evolutionary Structures (SAES), Layer Recombination (LR), Layer Mutation (LM), and Self-Adaptivity in Mutation (SAM). The evolution process starts with SAIP, and then it takes short and long steps to explore a variety of neural structures by SAES algorithm that uses both LR and LM algorithms to provide a variety of structures. Finally, the SAM algorithm guides the selected structures to a target structure. In this research, experiments applied EDNC on well-known tasks, including Facebook bAbI, copy, graph (shortest path), and 8-puzzle. The experiments show that the proposed method in both encodings automatically and without prior knowledge, produces an appropriate controller structure in the shortest time. It though shows that both proposed encodings reduce the evolution time by at least 73% in comparison with the baseline methods.

Ternary-task convolutional bidirectional neural turing machine for assessment of EEG-based cognitive workload

Cognitive workload plays a crucial role in the observation of mental activity which has great significance in brain-computer interfaces (BCI), cognitive neuroscience and biomedical fields. The estimation of cognitive state based on the classification of the electroencephalograph (EEG) is a hot issue received more and more attention. So far, a variety of Deep Learning models have been raised, which has yielded improvements in feature extraction and classification. However, the existing models reveal shortcomings in processing spatial, spectral, and temporal features of the EEG. In this paper, we propose a deep hybrid Network called Ternary-task Convolutional Bidirectional Neural Turing Machine (TT-CBNTM) to perform cognitive state assessment. First, TT-CBNTM consists of Convolutional Neural Network (CNN) and Bidirectional Neural Turing Machine (BNTM), of which CNN is applied to preserve the spatial and spectral representations of EEG, while the BNTM is applied to learn temporal representations from features which are extracted from the CNN. Second, we propose a new strategy called ternary-task regularization framework to induce the overfitting on the EEG database. The main task is to assess EEG-based cognitive workload through classification of EEG signals. The auxiliary tasks is identification and verification, through which we can increase the inter-class variations and reduce the intra-class variations, further lead to a better result of the EEG-based cognitive workload classification. The classication accuracy of our TT-CBNTM is 96.3 %, yielding 5 % improvement over the state-of-the-art models. This demonstrates the significant effectiveness of our approach which can be applied successfully to cognitive monitoring systems.