Abstract
Long short-term memory (LSTM) has shown good performance when used with sequential data, but gradient vanishing or exploding problem can arise, especially when using deeper layers to solve complex problems. Thus, in this paper, we propose a new LSTM cell termed long short-time complex memory (LSTCM) that applies an activation function to the cell state instead of a hidden state for better convergence in deep layers. Moreover, we propose a sinusoidal function as an activation function for LSTM and the proposed LSTCM instead of a hyperbolic tangent activation function. The performance capabilities of the proposed LSTCM cell and the sinusoidal activation function are demonstrated through experiments on various natural language benchmark datasets, in this case the Penn Tree-bank, IWSLT 2015 English-Vietnamese, and WMT 2014 English-German datasets.
Highlights
R ECENTLY, deep learning approaches including feed-forward networks, convolution neural networks (CNNs), and recurrent neural networks (RNNs) have shown good performance in many fields
On a language modeling task based on the Penn Treebank (PTB) dataset, the difference between the perplexity level between the training and the test datasets was less when the independently long shortterm memory (ILSTM) and ILSTCM were applied as compared to when long short-term memory (LSTM) and long short-time complex memory (LSTCM) were applied, meaning that the independent concept prevents the network overfitting problem
This paper proposed what is termed a long short-time complex memory (LSTCM) cell to solve the gradient vanishing problem in recurrent neural networks (RNNs) and long shortterm memory (LSTM), especially when the network is deep
Summary
R ECENTLY, deep learning approaches including feed-forward networks, convolution neural networks (CNNs), and recurrent neural networks (RNNs) have shown good performance in many fields. The basic approach to solving complex problems with deep learning is to create a deeper network or a more complex network This is true, in RNN research; i.e., the stacking of multiple recurrent layers or the use of more complex cells, such as long short-term memory (LSTM) [10], gated recurrent unit (GRU) [11] and neural architecture search (NAS) [12] cells. The proposed cell is referred to as long shorttime complex memory (LSTCM) With this new activation technique, the proposed LSTCM cell reduces the gradient vanishing problem in the layers, creating and training a deeper network for complex problems.
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
