Time-series prediction using a regularized self-organizing long short-term memory neural network

Abstract

Industrial process data are naturally in the form of complex time-series with high nonlinearities and dynamics. Long short-term memory (LSTM) networks are suitable for developing prediction models to handle nonlinear and dynamic process. However, LSTM neural networks have typically large and predefined structures, which may result in overfitting, and an optimal hidden neurons for a given problem cannot be automatically obtained. For this reason, a regularized self-organizing LSTM (RSO-LSTM) is proposed to optimize both the structure and the parameters of the network. First, an adaptive learning algorithm based on l2-norm regularization is introduced for parameter adjustment. Thereafter, both the prediction accuracy and weight dispersion are considered to avoid overfitting. Second, a growing strategy is designed based on hidden neuronal sensitivity. The structure of the LSTM can then be determined automatically with improved compactness. Finally, a convergence analysis is performed to ensure the feasibility of RSO-LSTM. To demonstrate the merits of the proposed RSO-LSTM for time-series​ prediction, its results for three benchmark experiments and real industrial data of a municipal solid waste incineration process were examined and compared with those of other methods. The results indicated the superiority and potential of RSO-LSTM for industrial applications.