Sparse LSTM neural network with hybrid PSO algorithm

Abstract

As an improved recurrent neural network, Long short-term memory (LSTM) neural network have been widely applied in many areas. However, redundant resources of calculation and memory are often required in the dense LSTM neural network and the over-fitting problem can even be caused. It hinders the practical application of network. To enhance the sparsity and generalization ability, the sparse LSTM neural network with hybrid particle swarm optimization algorithm (SLSTM-HPSO) is proposed. Firstly, based on LSTM, the hybrid coding method is established to directly denote the state and value of network weights. Secondly, the fitness function which is composed of the network training error and ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</inf> norm term is introduced to control the accuracy and sparsity of network, simultaneously. Thirdly, the update strategy is designed according to the coding method and fitness function to search for optimal solution. Finally, the proposed method is respectively 47.72% and 47.02% better than traditional LSTM neural network in terms of prediction accuracy and sparsity.