Direct model of memory properties and the linear reservoir topologies in echo state networks

Abstract

Echo state networks (ESN), with a large number of randomly connected neurons (called “reservoir”) and a simple linear output neuron, are a kind of novel recurrent neural network (RNN). One of the most important properties of ESN is short-term memory (STM), which is indispensable for time varying information processing. However, due to the random connection of neurons in the reservoir, the relationship between the topological structure of the reservoir and STM in ESN is not fully understood. In this paper, we concentrate on ESN with a linear reservoir, which consists of neurons with identity activation functions. We transform the iterative mathematical model of ESN into a direct one. In the model, we establish a direct relationship between the memory capacity of ESN and its connectivity, which can obtain the reservoir topology through STM in ESN. We find that some reservoir topologies proposed by previous papers are the special solutions of our method. Furthermore, the proposed model can provide an effective way to adjust the reservoir to a state near the edge of chaos, where previous studies have shown that the reservoir can achieve maximum computational capabilities.