Echo state network with logistic mapping and bias dropout for time series prediction

Abstract

An echo state network (ESN) is a special structure of a recurrent neural network in which the recurrent neurons are randomly connected. ESN models that have achieved high accuracy on time series prediction tasks can be utilized as time series prediction models in many fields. Nevertheless, in most ESN models, the input weights are irregularly generated and the reservoir layer units are generally redundant, which cannot guarantee that the ESN models will always be optimal for a given task. In this paper, a novel ESN model that combines logistic mapping (LM) and bias dropout (BD) algorithms is proposed to optimize the irregular input weight matrix and generate a superior and simpler reservoir. Initially, the initial input weight matrix of ESN is replaced by an LM input weight matrix that is generated by the recurrent LM algorithm. Meanwhile, the reservoir, which can convert the input space into a high-dimensional feature space, is formed by the input signals and the LM input weight matrix. Then, the units with low activation values that are determined by a reservoir dropout probability are discarded through the BD algorithm. The dropout probability is determined by the contributions of the reservoir units to the training performance. Three multivariable benchmark tasks and four univariate real-world time series tasks indicate that the proposed LM-BD-ESN model is effective in reducing the testing time, reservoir size, and model complexity while improving the performance of the traditional ESN.