Chain-Structure Echo State Network With Stochastic Optimization: Methodology and Application.

Abstract

In this article, a chain-structure echo state network (CESN) with stacked subnetwork modules is newly proposed as a new kind of deep recurrent neural network for multivariate time series prediction. Motivated by the philosophy of "divide and conquer," the related input vectors are first divided into clusters, and the final output results of CESN are then integrated by successively learning the predicted values of each clustered variable. Network structure, mathematical model, training mechanism, and stability analysis are, respectively, studied for the proposed CESN. In the training stage, least-squares regression is first used to pretrain the output weights in a module-by-module way, and stochastic local search (SLS) is developed to fine-tune network weights toward global optima. The loss function of CESN can be effectively reduced by SLS. To avoid overfitting, the optimization process is stopped when the validation error starts to increase. Finally, SLS-CESN is evaluated in chaos prediction benchmarks and real applications. Four different examples are given to verify the effectiveness and robustness of CESN and SLS-CESN.