Abstract
An increasing number of neural memory networks have been developed, leading to the need for a systematic approach to analyze and compare their underlying memory structures. Thus, in this paper, we first create a framework for memory organization and then compare four popular dynamic models: vanilla recurrent neural network, long short-term memory, neural stack, and neural RAM. This analysis helps to open the dynamic neural networks' black box from the memory usage prospective. Accordingly, a taxonomy for these networks and their variants is proposed and proved using a unifying architecture. With the taxonomy, both network architectures and learning tasks are classified into four classes, and a one-to-one mapping is built between them to help practitioners select the appropriate architecture. To exemplify each task type, four synthetic tasks with different memory requirements are selected. Moreover, we use some signal processing applications and two natural language processing applications to evaluate the methodology in a realistic setting.
Highlights
Memory has a pivotal role in human cognition and many different types are well known and intensively studied[1]
Our conclusion is that there is an hierarchical organization in the sense that each one of them can be seen as a special case of another one given the following order, i.e. vanilla recurrent neural networks (RNN)⊆ Long short term memory (LSTM)⊆neural stack⊆neural RAM
We will prove neural stack is a special case of neural RAM, LSTM is a special case of neural stack and vanilla RNN is a special case of LSTM
Summary
Abstract—In this paper, a taxonomy for memory networks is proposed based on their memory organization. The taxonomy includes all the popular memory networks: vanilla recurrent neural network (RNN), long short term memory (LSTM ), neural stack and neural Turing machine and their variants. The taxonomy puts all these networks under a single umbrella and shows their relative expressive power , i.e. vanilla RNN⊆ LSTM⊆neural stack⊆neural RAM. The differences and commonality between these networks are analyzed. These differences are connected to the requirements of different tasks which can give the user instructions of how to choose or design an appropriate memory network for a specific task. We use two natural language processing problems to discuss how this taxonomy helps choosing the appropriate neural memory networks for real world problem
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callMore From: IEEE Transactions on Neural Networks and Learning Systems
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
