Deep echo state network with reservoirs of multiple activation functions for time-series prediction

Abstract

In this paper, an improved deep echo state network is proposed, named as multiple activation functions deep echo state network (MAF-DESN), where states are activated by multiple activation functions. A sufficient condition for MAF-DESN is given to guarantee that MAF-DESN possesses the echo state property. Finally, the MAF-DESN is applied to chaotic time-series predictions and compared to other ESN deformation models and popular LSTM. Simulation results show that under same network size condition, MAF-DESN possesses stronger explanatory power in chaotic far-infrared laser predictions (R-square=0.9537, others≤0.6487), and better fitting ability in daily foreign exchange rates (MAE=0.0040, others≥0.0047) and chaotic far-infrared laser (MAE=3.4042, others≥4.9021). In high-dimension-input task, MAF-DESN improved the performance when the results were compared (R-square=0.4274, others≤0.3975 and MAE=5.2221, others≥7.6876), while the train time of MAF-DESN did not increase when compared to DESN.