Echo State Network Model Research Articles

Bayesian Optimized Echo State Network Applied to Short-Term Load Forecasting

Load forecasting impacts directly financial returns and information in electrical systems planning. A promising approach to load forecasting is the Echo State Network (ESN), a recurrent neural network for the processing of temporal dependencies. The low computational cost and powerful performance of ESN make it widely used in a range of applications including forecasting tasks and nonlinear modeling. This paper presents a Bayesian optimization algorithm (BOA) of ESN hyperparameters in load forecasting with its main contributions including helping the selection of optimization algorithms for tuning ESN to solve real-world forecasting problems, as well as the evaluation of the performance of Bayesian optimization with different acquisition function settings. For this purpose, the ESN hyperparameters were set as variables to be optimized. Then, the adopted BOA employs a probabilist model using Gaussian process to find the best set of ESN hyperparameters using three different options of acquisition function and a surrogate utility function. Finally, the optimized hyperparameters are used by the ESN for predictions. Two datasets have been used to test the effectiveness of the proposed forecasting ESN model using BOA approaches, one from Poland and another from Brazil. The results of optimization statistics, convergence curves, execution time profile, and the hyperparameters’ best solution frequencies indicate that each problem requires a different setting for the BOA. Simulation results are promising in terms of short-term load forecasting quality and low error predictions may be achieved, given the correct options settings are used. Furthermore, since there is not an optimal global optimization solution known for real-world problems, correlations among certain values of hyperparameters are useful to guide the selection of such a solution.

Destination prediction using deep echo state network

Final destination prediction based on initial partial trajectories plays an important role in location-based services (LBSs) and urban computing. Several studies on destination prediction make use of extra information which is often unavailable and reveal privacy of data owners. Many existing approaches are based on various Markov chain models, while they have significant drawbacks due to the Markov property. In general, the solution based on neural networks performs much better than the above approaches, in which Recurrent Neural Networks (RNNs) provide an effective temporal processing capacity for destination prediction. The Echo State Network (ESN) is the state-of-the-art approach for the design of efficiently trained RNNs, which provides an architecture and supervised learning principle. In this work, we adapt the deep Echo State Network (deepESN) model to taxi destination prediction, which uses only historic trajectories and achieves good results. Based on the original deepESN, we propose a novel model called deepESN with Dual Input (deepESN-DI) to optimize the time complexity of deepESN in training with large amount of data, and combine the Reservoir Computing (RC) with Multi-Layer Perceptron (MLP) to improve the prediction accuracy, which achieves better prediction results. Experimental results using the dataset provided by the Kaggle taxi trajectory prediction challenge show that our deepESN-DI and deepESN-DI-MLP approaches outperform other destination prediction models based on neural networks, namely MLP, RNN and the shallow ESN architectures.

Echo state network optimization using binary grey wolf algorithm

The echo state network (ESN) is a powerful recurrent neural network for time series modelling. ESN inherits the simplified structure and relatively straightforward training process of conventional neural networks, and shows strong computational capabilities to solve nonlinear problems. It is able to map low-dimensional input signals to high-dimensional space for information extraction, but it is found that not every dimension of the reservoir output directly contributes to the model generalization. This work aims to improve the generalization capabilities of the ESN model by reducing the redundant reservoir output features. A novel hybrid model, namely binary grey wolf echo state network (BGWO-ESN), is proposed which optimises the ESN output connection by the feature selection scheme. Specially, the feature selection scheme of BGWO is developed to improve the ESN output connection structure. The proposed method is evaluated using synthetic and financial data sets. Experimental results demonstrate that the proposed BGWO-ESN model is more effective than other benchmarks, and obtains the lowest generalization error.

Echo State Networks with Self-Normalizing Activations on the Hyper-Sphere

Among the various architectures of Recurrent Neural Networks, Echo State Networks (ESNs) emerged due to their simplified and inexpensive training procedure. These networks are known to be sensitive to the setting of hyper-parameters, which critically affect their behavior. Results show that their performance is usually maximized in a narrow region of hyper-parameter space called edge of criticality. Finding such a region requires searching in hyper-parameter space in a sensible way: hyper-parameter configurations marginally outside such a region might yield networks exhibiting fully developed chaos, hence producing unreliable computations. The performance gain due to optimizing hyper-parameters can be studied by considering the memory–nonlinearity trade-off, i.e., the fact that increasing the nonlinear behavior of the network degrades its ability to remember past inputs, and vice-versa. In this paper, we propose a model of ESNs that eliminates critical dependence on hyper-parameters, resulting in networks that provably cannot enter a chaotic regime and, at the same time, denotes nonlinear behavior in phase space characterized by a large memory of past inputs, comparable to the one of linear networks. Our contribution is supported by experiments corroborating our theoretical findings, showing that the proposed model displays dynamics that are rich-enough to approximate many common nonlinear systems used for benchmarking.

Chaotic Time Series Prediction Using a Novel Echo State Network Model with Input Reconstruction, Bayesian Ridge Regression and Independent Component Analysis

This paper presents a novel Echo State Network (ESN) model for chaotic time series prediction, which consists of three steps including input reconstruction, dimensionality reduction and regression. First, phase-space reconstruction is used to reconstruct the original ‘attractor’ of the input time series. Then, Independent Component Analysis (ICA) is used to identify independent components, reduce dimensionality and overcome multicollinearity problem of the reconstructed input matrix. Finally, Bayesian Ridge Regression provides accurate predictions thanks to its regularization effect to avoid over-fitting and its robustness to noise owing to its probabilistic strategy. Our experimental results show that our model significantly outperforms other ESN models in predicting both artificial and real-world chaotic time series.

An echo state network architecture based on quantum logic gate and its optimization

Quantum neural network (QNN) is developed based on two classical theories of quantum computation and artificial neural networks. It has been proved that quantum computing is an important candidate for improving the performance of traditional neural networks. In this work, inspired by the QNN, the quantum computation method is combined with the echo state networks (ESNs), and a hybrid model namely quantum echo state network (QESN) is proposed. Firstly, the input training data is converted to quantum state, and the internal neurons in the dynamic reservoir of ESN are replaced by qubit neurons. Then in order to maintain the stability of QESN, the particle swarm optimization (PSO) is applied to the model for the parameter optimizations. The synthetic time series and real financial application datasets (Standard & Poor's 500 index and foreign exchange) are used for performance evaluations, where the ESN, autoregressive integrated moving average (ARIMAX) are used as the benchmarks. Results show that the proposed PSO-QESN model achieves a good performance for the time series predication tasks and is better than the benchmarking algorithms. Thus, it is feasible to apply quantum computing to the ESN model, which provides a novel method to improve the ESN performance.

Sample-level sound synthesis with recurrent neural networks and conceptors.

Conceptors are a recent development in the field of reservoir computing; they can be used to influence the dynamics of recurrent neural networks (RNNs), enabling generation of arbitrary patterns based on training data. Conceptors allow interpolation and extrapolation between patterns, and also provide a system of boolean logic for combining patterns together. Generation and manipulation of arbitrary patterns using conceptors has significant potential as a sound synthesis method for applications in computer music but has yet to be explored. Conceptors are untested with the generation of multi-timbre audio patterns, and little testing has been done on scalability to longer patterns required for audio. A novel method of sound synthesis based on conceptors is introduced. Conceptular Synthesis is based on granular synthesis; sets of conceptors are trained to recall varying patterns from a single RNN, then a runtime mechanism switches between them, generating short patterns which are recombined into a longer sound. The quality of sound resynthesis using this technique is experimentally evaluated. Conceptor models are shown to resynthesise audio with a comparable quality to a close equivalent technique using echo state networks with stored patterns and output feedback. Conceptor models are also shown to excel in their malleability and potential for creative sound manipulation, in comparison to echo state network models which tend to fail when the same manipulations are applied. Examples are given demonstrating creative sonic possibilities, by exploiting conceptor pattern morphing, boolean conceptor logic and manipulation of RNN dynamics. Limitations of conceptor models are revealed with regards to reproduction quality, and pragmatic limitations are also shown, where rises in computation and memory requirements preclude the use of these models for training with longer sound samples. The techniques presented here represent an initial exploration of the sound synthesis potential of conceptors, demonstrating possible creative applications in sound design; future possibilities and research questions are outlined.

Evolving Local Plasticity Rules for Synergistic Learning in Echo State Networks.

Existing synaptic plasticity rules for optimizing the connections between neurons within the reservoir of echo state networks (ESNs) remain to be global in that the same type of plasticity rule with the same parameters is applied to all neurons. However, this is biologically implausible and practically inflexible for learning the structures in the input signals, thereby limiting the learning performance of ESNs. In this paper, we propose to use local plasticity rules that allow different neurons to use different types of plasticity rules and different parameters, which are achieved by optimizing the parameters of the local plasticity rules using the evolution strategy (ES) with covariance matrix adaptation (CMA-ES). We show that evolving neural plasticity will result in a synergistic learning of different plasticity rules, which plays an important role in improving the learning performance. Meanwhile, we show that the local plasticity rules can effectively alleviate synaptic interferences in learning the structure in sensory inputs. The proposed local plasticity rules are compared with a number of the state-of-the-art ESN models and the canonical ESN using a global plasticity rule on a set of widely used prediction and classification benchmark problems to demonstrate its competitive learning performance.

Short-term wind speed forecasting based on autoregressive moving average with echo state network compensation

In order to improve the forecasting accuracy of short-term wind speed, a forecasting method based on autoregressive moving average with echo state network compensation is proposed in this article. First, the linear and nonlinear characteristics of short-term wind speed can be determined by Brock–Dechert–Scheinkman statistics method. Then, autoregressive moving average model is used for modeling and to forecast the linear component of short-term wind speed. The linear component of short-term wind speed sequence is obtained. Artificial bee colony algorithm–optimized echo state network model is used as the forecasting model of forecasting error sequences with the nonlinear characteristic. Finally, the final forecasting value is obtained by adding forecasting values of autoregressive moving average model and forecasting error values of echo state network model. k-fold cross-validation is used to improve the generalization ability of the forecasting model. The simulation comparison results show that the proposed forecasting method has higher prediction accuracy with the smaller prediction error. The forecasting indicators are also better than other forecasting methods.

Time Series Prediction Model of Grey Wolf Optimized Echo State Network

As a novel recursion neural network, Echo State Networks (ESN) are characterized by strong nonlinear prediction capability and effective and straightforward training algorithms. However, conventional ESN predictions require a large volume of training samples. Meanwhile, the time sequence data are complicated and unstable, resulting in insufficient learning of this network and difficult training. As a result, the accuracies of conventional ESN predictions are limited. Aimed at this issue, a time series prediction model of Grey Wolf optimized ESN has been proposed. Wout of ESN was optimized using the Grey Wolf algorithm and predictions of time series data were achieved using simplified training. The results indicated that the optimized time series prediction method exhibits superior prediction accuracy at a small sample size, compared with conventional prediction methods.

Echo state networks regulated by local intrinsic plasticity rules for regression

Intrinsic plasticity, as a biologically inspired unsupervised learning rule, is used for adapting the intrinsic excitability of the reservoir neurons. Existing intrinsic plasticity rules can only select a set of fixed rule parameters for the whole reservoir neurons, which affects the learning performance due to the lack of flexibility in providing intrinsic plasticity. In this paper, we present an echo state network (ESN) with local intrinsic plasticity rule built by different reservoir neurons which can adopt the intrinsic plasticity rule with different rule parameters to adjust its intrinsic excitability. And the covariance matrix adaptation evolution strategy is used to search and select the rule parameters corresponding to different reservoir neurons. Compared with several state-of-the-art ESN models and an ESN with the global plasticity rule, the proposed local intrinsic plasticity rule is able to achieve much better performance in some benchmark prediction tasks.

Demystifying echo state network with deterministic simple topologies

Echo state networks (ESN) are a special type of recurrent neural networks (RNN) with distinct performance in the field of reservoir computing (RC). The state space of the ESN is initially randomised and the reservoir weights are fixed with training done only on the state readout. Beside the advantages of ESN, there remains some opacity in the dynamic properties of the reservoir due to the presence of randomisation. Our aim in this paper is to demystify the model of ESN in a complete deterministic structure with the use of different proposed reservoir structures (topologies) and compare their performance with the random ESN on different benchmark datasets. All applied topologies maintain the simplicity of random ESN computation complexity. Most of the topologies showed comparable or even better performance.

Deep Reservoir Neural Networks for Trees

Tree structured data are a flexible tool to properly express many forms of hierarchical information. However, learning of such data through deep recursive models is particularly demanding. We will show through the introduction of the Deep Tree Echo State Network model (DeepTESN) that the randomized Neural Networks framework offers a formidable approach to allow an efficient treatment of learning in tree structured domains by deep architectures. Theoretical properties, for the Reservoir Computing setup constraints, and empirical behavior of the proposed approach are analyzed, showing its feasibility and accuracy.

Deep echo state networks with uncertainty quantification for spatio‐temporal forecasting

AbstractLong‐lead forecasting for spatio‐temporal systems can entail complex nonlinear dynamics that are difficult to specify a priori. Current statistical methodologies for modeling these processes are often highly parameterized and, thus, challenging to implement from a computational perspective. One potential parsimonious solution to this problem is a method from the dynamical systems and engineering literature referred to as an echo state network (ESN). ESN models use reservoir computing to efficiently compute recurrent neural network forecasts. Moreover, multilevel (deep) hierarchical models have recently been shown to be successful at predicting high‐dimensional complex nonlinear processes, particularly those with multiple spatial and temporal scales of variability (such as those we often find in spatio‐temporal environmental data). Here, we introduce a deep ensemble ESN (D‐EESN) model. Despite the incorporation of a deep structure, the presented model is computationally efficient. We present two versions of this model for spatio‐temporal processes that produce forecasts and associated measures of uncertainty. The first approach utilizes a bootstrap ensemble framework, and the second is developed within a hierarchical Bayesian framework (BD‐EESN). This more general hierarchical Bayesian framework naturally accommodates non‐Gaussian data types and multiple levels of uncertainties. The methodology is first applied to a data set simulated from a novel non‐Gaussian multiscale Lorenz‐96 dynamical system simulation model and, then, to a long‐lead United States (U.S.) soil moisture forecasting application. Across both applications, the proposed methodology improves upon existing methods in terms of both forecast accuracy and quantifying uncertainty.

Research on Sparsity of Output Synapses in Echo State Networks

This paper presents an improved model of echo state networks (ESNs) and gives the definitions of energy consumption, energy efficiency, etc. We verify the existence of redundant output synaptic connections by numerical simulations. We investigate the relationships among energy consumption, prediction step, and the sparsity of ESN. At the same time, the energy efficiency and the prediction steps are found to present the same variation trend when silencing different synapses. Thus, we propose a computationally efficient method to locate redundant output synapses based on energy efficiency of ESN. We find that the neuron states of redundant synapses can be linearly represented by the states of other neurons. We investigate the contributions of redundant and core output synapses to the performance of network prediction. For the prediction task of chaotic time series, the predictive performance of ESN is improved about hundreds of steps by silencing redundant synapses.

A prognostic methodology for power MOSFETs under thermal stress using echo state network and particle filter

Reinforcing the reliability of power semiconductor devices is crucial for extending the lifetime of the power-converter based electrical systems. This paper aims at developing a novel prognostics methodology for estimating the Remaining Useful Life (RUL) of the power Metal-Oxide Field-Effect Transistors (MOSFETs). The variation of on-state resistance as an important fault indicator under thermal overstress is utilized as the main database. A recently proposed neural network paradigm, namely Echo State Network (ESN) is utilized here to derive a degradation model, taking into account its high efficiency in modeling nonlinear dynamical systems. Meanwhile, a particle filter approach is developed to update the initially trained ESN model and to quantify the uncertainty of the RUL prediction online. The accuracy and efficiency of the proposed prognostic methodology has been verified based on an accelerated aging experimental dataset.

Dynamical regularized echo state network for time series prediction

Echo state networks (ESNs) have been widely used in the field of time series prediction. However, it is difficult to automatically determine the structure of ESN for a given task. To solve this problem, the dynamical regularized ESN (DRESN) is proposed. Different from other growing ESNs whose existing architectures are fixed when new reservoir nodes are added, the current component of DRESN may be replaced by the newly generated network with more compact structure and better prediction performance. Moreover, the values of output weights in DRESN are updated by the error minimization-based method, and the norms of output weights are controlled by the regularization technique to prevent the ill-posed problem. Furthermore, the convergence analysis of the DRESN is given theoretically and experimentally. Simulation results demonstrate that the proposed approach can have few reservoir nodes and better prediction accuracy than other existing ESN models.

Estimate and characterize PV power at demand-side hybrid system

Power forecasting, in a hybrid photovoltaic (PV) system, is an important issue regarding to the control and optimization of energy systems. In this work, multi-clustered echo state network (MCESN) models are proposed to directly perform the forecast of PV power generation. Furthermore, data characteristics of measured and estimated PV power are qualitatively investigated via data mining approaches. These characteristics include seasonality, stationarity (or non-stationarity) and complexity analysis. Simulation results indicate that the proposed MCESN model is able to precisely forecast PV power one-hour-ahead. The performance on the 24-h-ahead forecast is competitive with the correlation coefficient 99% for sunny days, and 91–98% for cloudy days. Results of data analysis unveil that critical characteristics between the measured and estimated PV power data are analogous. Comparison studies also show that MCESN could achieve more accurate prediction, compared with auto-regressive moving average (ARMA), back propagation (BP) neural networks.

Nonlinear Model Predictive Control of an Oil Well with Echo State Networks

In oil production platforms, processes are nonlinear and prone to modeling errors, as the flow regime and components are not entirely known and can bring about structural uncertainties, making the design of predictive control algorithms a challenge. In this work, an efficient data-driven framework for Model Predictive Control (MPC) using Echo State Networks (ESN) as the prediction model is proposed. Unlike previous works, the ESN model for MPC is only linearized partially: while the free response of the system is kept fully nonlinear, only the forced response is linearized. This MPC framework is known in the literature as the Practical Nonlinear Model Predictive Controller (PNMPC). In this work, by using the analytically computed gradient from the ESN model, a finite difference method is not needed to compute derivatives as in PNMPC. The proposed method, called PNMPC-ESN, is applied to control a simplified model of a gas-lifted oil well, managing to successfully control the plant, obeying the established constraints while maintaining setpoint tracking.

Prediction of component content in rare earth extraction process based on ESNs-Adaboost

Aiming at the problem that the component content of elements in REEP (rare earth extraction process) is difficult to detect, a modeling method for component content in REEP based on ESNs-Adaboost is proposed. It uses ESN (echo state network) with fast training speed and high stability to establish multiple identification model of REEP. And then combined with the improved Adaboost algorithm, multiple models are integrated into a final ESNs-Adaboost model of component content of REEP according to the combination rule. The improved Adaboost algorithm has an advantage that its threshold can be adjusted adaptively with the training error. By collecting data from CePr/Nd extraction process, the simulation results show that the proposed method has high prediction accuracy, strong generalization ability, good robustness and better performance than single ESN model, which can meet the requirements of rapid detection of component content in extraction field.