Echo state networks with decoupled reservoir states

Abstract

Echo state networks (ESNs) are a novel form of recurrent neural networks that provide an efficient and powerful computational model to approximate dynamic nonlinear systems. Why a random, large, fixed recurrent neural network (reservoir) has such astonishing performance in approximating nonlinear systems remains a mystery. In this paper, we first compare two reservoir scenarios in ESNs, i.e. sparsely versus fully connected reservoirs, and show that the eigenvalues of these reservoir weight matrices have the same limit distribution in the complex plane. We discuss the link between the eigenvalues of the reservoir weight matrix and the ESN approximation ability in a simplified ESN case. We propose a new ESN with decoupled reservoir states, in which the neurons in the reservoir are decoupled into single or pairs of neurons. A reservoir state back-elimination strategy is presented, which not only reduces model complexity but also increases numerical stability when calculating the output weights. The proposed model is tested in a communication channel equalization problem and applied to gene expression time series modeling with very promising results.