Optimization of Echo State Networks by Covariance Matrix Adaption Evolutionary Strategy

Abstract

Echo state networks (ESNs) have been shown to be an effective alternative to conventional recurrent neural networks due to its simple training process and good fitting performance of time series modelling tasks. In the primary ESN principle, the random setting of reservoir is considered to be the main advantage of ESN. However, because of the randomly generated connectivity and weight parameters, appropriate setting of the structural parameters which can significantly influence the modelling accuracy is considered a key issue in building ESN models. Evolutionary Strategy (ES) has been shown being a powerful stochastic global optimization method. Moreover, covariance matrix adaption evolutionary strategy (CMA-ES) is an artistically and parallel search method which transforms the searching covariance matrix to guide the best search direction. This paper proposes a CMA-ES-ESN method to optimize several structural parameters of an ESN such as reservoir size, leak rate and spectral radius factor. Finally, the results are compared with those from the original ESN and GA-ESN, ESN optimized by genetic algorithm.