Deep Echo State Network Research Articles

Cooperative co-evolution deep echo state network for time series prediction

Abstract The deep echo state network demonstrates strong predictive performance in time series tasks, but inter-layer weight optimization and reservoir structure design are still challenging tasks. To address this, we propose the cooperative co-evolutionary deep echo state network (CCEDESN). Firstly, inspired by the hierarchical structure of the brain, we introduce a modular hierarchical design. Each layer of the reservoir contains multiple sub-reservoirs, with central neurons in each sub-reservoir interacting with those in other sub-reservoirs to enhance information processing. Secondly, an advanced cooperative co-evolutionary algorithm is proposed to simultaneously optimize the layer connection weights and hyperparameters of the deep echo state network. We evaluate the CCEDESN model on the Mackey-Glass system, sunspot data, and Apple stock opening prices. The experimental results show that the CCEDESN model achieves superior prediction accuracy compared to other models.

A spatial reconstitution model based on geographic deep echo state networks for ionospheric foF2 in East Asia

A spatial reconstitution model based on geographic deep echo state networks for ionospheric foF2 in East Asia

Expression of Concern: Grey wolf optimization–based deep echo state network for time series prediction

Expression of Concern: Grey wolf optimization–based deep echo state network for time series prediction

Deep echo state network with projection-encoding for multi-step time series prediction

To fully utilize the advantage of reservoir computing in deep network modeling, a deep echo state network with projection-encoding (DEESN) is newly proposed for multi-step time series prediction in this paper. DEESN integrates multiple echo state network (ESN) modules and extreme learning machine (ELM) encoders in series arrays. Firstly, the kth ESN in DEESN learner is responsible for kth step ahead prediction. The forecast output and encoded reservoir states of the previous ESN module are concatenated with the input variable to form the new input signals for the next adjacent module. Therefore, the temporal dependency among future time steps can be learned, which contributes the performance improvement. Secondly, the ELM encoder is used to optimize the reservoir states for time consumption reduction. Finally, the effectiveness of DEESN is evaluated in artificial chaos benchmarks and real-world applications. Experimental results on six different datasets and comparative models demonstrate that the proposed DEESN has excellent accuracy and robust generalization for multi-step time series prediction.

Growing evolutional deep echo state network

Deep echo state network (Deep ESN) is a powerful method for time-series prediction, but developing the optimal structure of Deep ESN remains challenging. A novel generative framework, growing evolutional Deep ESN (GE Deep ESN) is proposed, where networks are generated through alternating growth and evolution. A model compression method is implemented during the evolution phase to optimize the inner structure of the reservoirs, and to automatically determine reservoir size. A simple termination criterion of growth is introduced to determine network depth. The feasibility and superiority of GE Deep ESN were validated experimentally with two benchmark prediction tasks. For real-time trajectory prediction in photoelectric tracking systems, an iterative learning model based on GE Deep ESN is proposed, the real time prediction result showed that network evolution enhances the model’s suitability for this task, and improves prediction precision.

Sparse compressed deep echo state network with improved arithmetic optimization algorithm for chaotic time series prediction

A sparse compressed deep echo state network (SCDESN) incorporating sparse input units, compressed sampling and principal component analysis (PCA) units is proposed in this paper, and theoretically proved the sufficient and necessary conditions to ensure the echo state characteristics of the proposed scheme. In addition, a hybrid arithmetic optimization algorithm based on matrix design strategy and boundary selection strategy (SVD-HAOA) was proposed to optimize the hyper-parameters of the SCDESN model, taking into account the characteristics of the SCDESN model. The steps and process for optimizing the hyper parameters of the SCDESN model using SVD-HAOA were presented. Finally, the SCDESN model optimized by SVD-HAOA was experimentally and summarized on a classic benchmark dataset and two real-world chaotic time series. The outcomes of the simulation indicated that the suggested model surpassed alternative models in performance and enhanced computational efficiency while maintaining prediction accuracy. This model may serve as a viable alternative for real-world applications.

Enhanced multi-step streamflow series forecasting using hybrid signal decomposition and optimized reservoir computing models

This study evaluates the use of different time series decomposition methods in association with echo state networks (ESNs), deep echo state networks (DeepESNs), and next generation reservoir computing (NGRC) models to predict the natural flow of water in two hydropower reservoirs with horizons of one, seven, fourteen, and twenty-one steps ahead. The Coyote optimization algorithm is used to adjust the hyperparameters of the ESNs. The use of reservoir computing (RC) methods with Coyote optimization algorithm, a swarm intelligence approach, is compared with the use of variational mode decomposition (VMD), empirical wavelet transform, and empirical mode decomposition. Analysis of the results shows that the use of signal decomposition methods in association with RC can improve the accuracy and performance of the prediction model. The results are compared using different statistical metrics, showing that, in most cases, decomposition methods can improve the forecasting performance of RC. Overall, NGRC outperformed ESN and DeepESN in most of evaluated cases. The best results were obtained using VMD for the forecast horizons of 7, 14, and 21 days ahead, with a decrease in mean absolute error in a range from 43.70% to 88.88% compared to all the other models. Using a Diebold–Mariano test, the results show that the use of VMD is statistically significant in all the cases, with a significant value of 1% in all tested cases of ESN forecasting.

Multi-scale dynamics by adjusting the leaking rate to enhance the performance of deep echo state networks.

The deep echo state network (Deep-ESN) architecture, which comprises a multi-layered reservoir layer, exhibits superior performance compared to conventional echo state networks (ESNs) owing to the divergent layer-specific time-scale responses in the Deep-ESN. Although researchers have attempted to use experimental trial-and-error grid searches and Bayesian optimization methods to adjust the hyperparameters, suitable guidelines for setting hyperparameters to adjust the time scale of the dynamics in each layer from the perspective of dynamical characteristics have not been established. In this context, we hypothesized that evaluating the dependence of the multi-time-scale dynamical response on the leaking rate as a typical hyperparameter of the time scale in each neuron would help to achieve a guideline for optimizing the hyperparameters of the Deep-ESN. First, we set several leaking rates for each layer of the Deep-ESN and performed multi-scale entropy (MSCE) analysis to analyze the impact of the leaking rate on the dynamics in each layer. Second, we performed layer-by-layer cross-correlation analysis between adjacent layers to elucidate the structural mechanisms to enhance the performance. As a result, an optimum task-specific leaking rate value for producing layer-specific multi-time-scale responses and a queue structure with layer-to-layer signal transmission delays for retaining past applied input enhance the Deep-ESN prediction performance. These findings can help to establish ideal design guidelines for setting the hyperparameters of Deep-ESNs.

Multi-scale deep echo state network for time series prediction

Multi-scale deep echo state network for time series prediction

Exploring deep echo state networks for image classification: a multi-reservoir approach

Echo state networks (ESNs) belong to the class of recurrent neural networks and have demonstrated robust performance in time series prediction tasks. In this study, we investigate the capability of different ESN architectures to capture spatial relationships in images without transforming them into temporal sequences. We begin with three pre-existing ESN-based architectures and enhance their design by incorporating multiple output layers, customising them for a classification task. Our investigation involves an examination of the behaviour of these modified networks, coupled with a comprehensive performance comparison against the baseline vanilla ESN architecture. Our experiments on the MNIST data set reveal that a network with multiple independent reservoirs working in parallel outperforms other ESN-based architectures for this task, achieving a classification accuracy of 98.43%. This improvement on the classical ESN architecture is accompanied by reduced training times. While the accuracy of ESN-based architectures lags behind that of convolutional neural network-based architectures, the significantly lower training times of ESNs with multiple reservoirs operating in parallel make them a compelling choice for learning spatial relationships in scenarios prioritising energy efficiency and rapid training. This multi-reservoir ESN architecture overcomes standard ESN limitations regarding memory requirements and training times for large networks, providing more accurate predictions than other ESN-based models. These findings contribute to a deeper understanding of the potential of ESNs as a tool for image classification.

Newman-Watts-Strogatz topology in deep echo state networks for speech emotion recognition

Cortical neural connectivity exhibits a small-world (SW) network topology. However, its role in neural information processing remains unclear. In this study, we investigate the learning performance of a Deep Echo State Network (DeepESN) that incorporates the Newman-Watts-Strogatz (NWS) graph topology as a reservoir. The short path length of the SW topology allows input information to reach output nodes faster, and the clustered arrangement of this topology can expand the range of the echo state property. Additionally, in this paper, we seek to identify the best set of these key factors, such as the spectral radius of the reservoir’s weight matrix and the hyperparameters neighbors' and probabilities. We evaluate the NWS-DeepESNs on four Speech Emotion Recognition (SER) tasks, which are RAVDESS, Emo-db, SAVEE, and TESS databases. By examining the NWS-DeepESN’s performance on SER tasks, we aim to determine the role of the NWS graph topology in enhancing neural information processing. The obtained results exhibited significant improvements in recognition accuracies (weighted and unweighted) with (76.45%, 75.79%), (87.89%, 87.14%), (99%, 98.62%), and (63%, 62.24%), respectively, for RAVDESS, Emo-db, SAVEE, and TESS databases. The proposed approach improves emotion recognition capabilities and yields more accurate results compared to existing state-of-the-art approaches.

Research on Credit Risk Prediction Method of Blockchain Applied to Supply Chain Finance

INTRODUCTION: From the perspective of blockchain, it establishes a credit risk evaluation index system for supply chain finance applicable to blockchain, constructs an accurate credit risk prediction model, and provides a reliable guarantee for the research of credit risk in supply chain finance.OBJECTIVES: To address the inefficiency of the current credit risk prediction and evaluation model for supply chain finance.METHODS: This paper proposes a combined blockchain supply chain financial credit risk prediction and evaluation method based on kernel principal component analysis and intelligent optimisation algorithm to improve Deep Echo State Network. Firstly, the evaluation system is constructed by describing the supply chain financial credit risk prediction and evaluation problem based on blockchain technology, analysing the evaluation indexes, and constructing the evaluation system; then, the parameters of DeepESN network are optimized by combining the kernel principal component analysis method with the JSO algorithm to construct the credit risk prediction and evaluation model of supply chain finance; finally, the effectiveness, robustness, and real-time performance of the proposed method are verified by simulation experiment analysis.RESULTS: The results show that the proposed method improves the prediction efficiency of the prediction model.CONCLUSION: The problems of insufficient scientific construction of index system and poor efficiency of risk prediction model of B2B E-commerce transaction size prediction method are effectively solved.

Performance Evaluation and Improvement of Deep Echo State Network Models in English Writing Assistance and Grammar Error Correctionn

INTRODUCTION: The research on the performance evaluation model of English writing tutoring and grammar error correction is very necessary, which is not only conducive to the rational allocation of teachers' writing tutoring resources, but also more conducive to the timely and effective correction of students' grammatical errors.OBJCTIVES: Aiming at the problems of non-specific quantification, low precision, and low real-time performance evaluation methods for English writing grammar error correction in current methods.METHODS: This paper proposes a grammar error correction performance evaluation method based on deep echo state network with gold rush optimisation algorithm. Firstly, by analysing the process of English writing assistance and grammatical error correction, we extract the evaluation features of grammatical error correction type and construct the performance evaluation system; then, we improve the deep confidence network through the gold rush optimization algorithm and construct the grammatical error correction performance evaluation model; finally, we analyse it through simulation experiments.RESULTS: The results show that the proposed method improves the evaluation accuracy, robustness. The absolute value of the relative error of the evaluation value of the syntactic error correction performance of the method is controlled within the range of 0.02.CONCLUSION: The problems of non-specific quantification, low precision and low real-time performance of the application of English writing grammar error correction performance assessment methods are solved.

Prediction for nonlinear time series by improved deep echo state network based on reservoir states reconstruction

With the aim to enhance prediction accuracy for nonlinear time series, this paper put forward an improved deep Echo State Network based on reservoir states reconstruction driven by a Self-Normalizing Activation (SNA) function as the replacement for the traditional Hyperbolic tangent activation function to reduce the model’s sensitivity to hyper-parameters. The Strategy was implemented in a two-state reconstruction process by first inputting the time series data to the model separately. Once, the time data passes through the reservoirs and is activated by the SNA activation function, the new state for the reservoirs is created. The state is input to the next layer, and the concatenate states module saves. Pairs of states are selected from the activated multi-layer reservoirs and input into the state reconstruction module. Multiple input states are transformed through the state reconstruction module and finally saved to the concatenate state module. Two evaluation metrics were used to benchmark against three other ESNs with SNA activation functions to achieve better prediction accuracy.

Design and Application of Adaptive Sparse Deep Echo State Network

Design and Application of Adaptive Sparse Deep Echo State Network

New energy power demand prediction and optimal scheduling based on artificial intelligence in smart grid

Abstract With the development of smart grid, the demand for new energy power increases. Improving the accuracy of new energy power demand forecast is an important basis for the orderly operation of power system. This article presents a new energy power demand forecasting method based on DESSA-NESN algorithm. First, differential evolution algorithm (DE) and sparrow search algorithm (SSA) are combined, and operations such as mutation, crossing and screening are introduced into the population updating process of SSA. The internal state function of the savings pool of the standard echo state network (ESN) is replaced by the hyperbolic tangent function to obtain the nonlinear echo state network (NESN). Then, the parameters of deep echo state network (DESN) are optimized using DESSA algorithm. The DESSA-DESN prediction model is established. Finally, the mean absolute percentage error (MAPE) and root mean square error (RMSE) of DESSA-NESN were 15.84 and 0.12%, respectively, and the prediction effect was better than other comparison models.

Continual Residual Reservoir Computing for Remaining Useful Life Prediction

In practical engineering applications, it is inevitable that the production is often faced with different working conditions. Therefore, it is necessary to have a continual learning system, which can adapt to a sequence of tasks and keep learning over the time. In this work, we propose a continual deep residual reservoir computing (DRRC) framework for the practical remaining useful life (RUL) prediction task. Specifically, we propose a novel deep echo state network structure with residual blocks to effectively mitigate the performance degradation of the deep reservoir computing framework and reduce the difficulty of model training. Further, the proposed framework is trained with the elastic weight consolidation (EWC) method to alleviate the impact of catastrophic forgetting in continual learning systems. Extensive experiments are conducted with the FEMTO-ST bearing and high intensity radiated fields (HIRF) battery dataset. And the proposed framework is proved to be effective in multiple continual learning tasks comparing with other state-of-the-art methods.

A Systematic Review of Echo State Networks From Design to Application

Recurrent Neural Network (RNN) has demonstrated its outstanding ability in sequence tasks and has achieved state-of-the-art in many applications, such as industrial and medical. Echo State Network (ESN) is a simple type of RNN and has emerged in the last decade as an alternative to gradient descent training-based RNN. ESN is practical, conceptually simple, and easy to implement with a strong theoretical ground. It can avoid non-converging and computationally expensive issues in gradient descent RNN methods. Since ESN was put forward in 2002, abundant existing works have promoted the progress of ESN, and the recently introduced deep ESN opened the way to uniting the merits of deep learning and reservoir computing. Besides, the combinations of ESNs with other machine learning models have also overperformed baselines in some applications. However, the apparent simplicity of ESNs can sometimes be deceptive. Successfully applying ESNs needs some experience. Thus, we reviewed over 300 related papers and provided a systematic overview for the first time. In this paper, we categorize the related methods into classical ESN, DeepESN, and combination. Then, we analyze them from the perspective of network designs and specific applications. Finally, we discuss the challenges and opportunities by proposing open problems and future work.

Calibrated forecasts of quasi‐periodic climate processes with deep echo state networks and penalized quantile regression

AbstractAmong the most relevant processes in the Earth system for human habitability are quasi‐periodic, ocean‐driven multi‐year events whose dynamics are currently incompletely characterized by physical models, and hence poorly predictable. This work aims at showing how (1) data‐driven, stochastic machine learning approaches provide an affordable yet flexible means to forecast these processes; (2) the associated uncertainty can be properly calibrated with fast ensemble‐based approaches. While the methodology introduced and discussed in this work pertains to synoptic scale events, the principle of augmenting incomplete or highly sensitive physical systems with data‐driven models to improve predictability is far more general and can be extended to environmental problems of any scale in time or space.

CT scan pancreatic cancer segmentation and classification using deep learning and the tunicate swarm algorithm.

Pancreatic cancer (PC) is a very lethal disease with a low survival rate, making timely and accurate diagnoses critical for successful treatment. PC classification in computed tomography (CT) scans is a vital task that aims to accurately discriminate between tumorous and non-tumorous pancreatic tissues. CT images provide detailed cross-sectional images of the pancreas, which allows oncologists and radiologists to analyse the characteristics and morphology of the tissue. Machine learning (ML) approaches, together with deep learning (DL) algorithms, are commonly explored to improve and automate the performance of PC classification in CT scans. DL algorithms, particularly convolutional neural networks (CNNs), are broadly utilized for medical image analysis tasks, involving segmentation and classification. This study explores the design of a tunicate swarm algorithm with deep learning-based pancreatic cancer segmentation and classification (TSADL-PCSC) technique on CT scans. The purpose of the TSADL-PCSC technique is to design an effectual and accurate model to improve the diagnostic performance of PC. To accomplish this, the TSADL-PCSC technique employs a W-Net segmentation approach to define the affected region on the CT scans. In addition, the TSADL-PCSC technique utilizes the GhostNet feature extractor to create a group of feature vectors. For PC classification, the deep echo state network (DESN) model is applied in this study. Finally, the hyperparameter tuning of the DESN approach occurs utilizing the TSA which assists in attaining improved classification performance. The experimental outcome of the TSADL-PCSC method was tested on a benchmark CT scan database. The obtained outcomes highlighted the significance of the TSADL-PCSC technique over other approaches to PC classification.