DeePr-ESN: A deep projection-encoding echo-state network

Abstract

As a recurrent neural network that requires no training, the reservoir computing (RC) model has attracted widespread attention in the last decade, especially in the context of time series prediction. However, most time series have a multiscale structure, which a single-hidden-layer RC model may have difficulty capturing. In this paper, we propose a novel multiple projection-encoding hierarchical reservoir computing framework called Deep Projection-encoding Echo State Network (DeePr-ESN). The most distinctive feature of our model is its ability to learn multiscale dynamics through stacked ESNs, connected via subspace projections. Specifically, when an input time series is projected into the high-dimensional echo-state space of a reservoir, a subsequent encoding layer (e.g., an autoencoder or PCA) projects the echo-state representations into a lower-dimensional feature space. These representations are the principal components of the echo-state representations, which removes the high frequency components of the representations. These can then be processed by another ESN through random connections. By using projection layers and encoding layers alternately, our DeePr-ESN can provide much more robust generalization performance than previous methods, and also fully takes advantage of the temporal kernel property of ESNs to encode the multiscale dynamics of time series. In our experiments, the DeePr-ESNs outperform both standard ESNs and existing hierarchical reservoir computing models on some artificial and real-world time series prediction tasks.