Echo State Network Research Articles

A double-cycle echo state network topology for time series prediction.

Echo state network (ESN) has gained wide acceptance in the field of time series prediction, relying on sufficiently complex reservoir connections to remember the historical features of the data and using these features to obtain the outputs by a simple linear readout. However, the randomness of its input and reservoir connections pose negative impacts on the prediction performance and performance stability of the models, the complexity of reservoir connections brings high time consumption during network computing, and the presence of randomness and complexity makes the hardware implementation of the ESN difficult. In response, we propose a double-cycle ESN (DCESN) based on the Li-ESN model, which has fixed weights to improve prediction performance and performance stability and simpler reservoir connections compared to the classical ESN to reduce the time consumption. The existence of both greatly reduces the difficulty of hardware implementation of the ESN and provides many conveniences for the future application of the ESN. Experimental results on many widely used time series datasets show that the DCESN has comparable or even better prediction performance than the ESN and good robustness against noise and parameter fluctuations.

Photovoltaic power prediction method for zero energy consumption buildings based on multi-feature fuzzy clustering and MAOA-ESN

Due to their high efficiency and compatibility with building integration, photovoltaic (PV) power generation systems are frequently utilized in zero energy buildings. However, in zero energy buildings with a high proportion of PV systems, the instability of the PV power generation has resulted in various issues, including a significant influence on the primary grid and inadequate photovoltaic power generation utilization. Accurately predicting PV power is crucial for buildings to utilize solar energy and achieve zero energy consumption fully. Consequently, this paper proposed a novel hybrid short-term PV power prediction method based on an echo state network, fuzzy clustering, similar day, and an intelligent optimization algorithm to improve the accuracy of PV power prediction for zero energy buildings. First, the dataset is partitioned using an improved fuzzy C-mean clustering method (FCM), effectively reducing the effect of PV diversity on the model. Subsequently, an improved similarity day algorithm is employed to select the training data of Echo State Networks (ESN) to minimize the effect of randomness. The improved similarly day algorithm reduces the effect of randomness by retaining valuable samples. Moreover, the echo state network (ESN) parameters are sought using a multi-strategy collaborative improved Archimedean optimization algorithm (MAOA) to avoid poor prediction due to improper parameter settings. Finally, the model is evaluated using historical Australian PV data. The results show that the method effectively reduces the impacts of stochasticity and nonlinearity in the PV power generation process of zero-energy buildings and increases the accuracy of zero-energy building PV power prediction.

Theoretical Foundation and Design Guideline for Reservoir Computing-Based MIMO-OFDM Symbol Detection

In this paper, we derive a theoretical upper bound on the generalization error of reservoir computing (RC), a special category of recurrent neural networks (RNNs). The specific RC implementation considered in this paper is the echo state network (ESN), and an upper bound on its generalization error is derived via the empirical Rademacher complexity (ERC) approach. While recent work in deriving risk bounds for RC frameworks makes use of a non-standard ERC measure and a direct application of its definition, our work uses the standard ERC measure and tools allowing fair comparison with conventional RNNs. The derived result shows that the generalization error bound obtained for ESNs is tighter than the existing bound for vanilla RNNs, suggesting easier generalization for ESNs. With the ESN applied to symbol detection in MIMO-OFDM (Multiple Input Multiple Output-Orthogonal Frequency Division Multiplexing) systems, we show how the derived generalization error bound can guide underlying system design. Specifically, the derived bound together with the empirically characterized training loss is utilized to identify the optimum reservoir size in neurons for the ESN-based symbol detector. Finally, we corroborate our theoretical findings with results from simulations that employ 3GPP standards-compliant wireless channels, signifying the practical relevance of our work.

Remaining useful life prediction method of lithium-ion batteries is based on variational modal decomposition and deep learning integrated approach

A common method based on variational modal decomposition (VMD) and an integrated depth model is proposed to address the problem that it is difficult to precisely anticipate the remaining useful life (RUL) of lithium-ion batteries (LIBs). Initially, VMD is employed to decompose the LIBs capacity data in multiple scales to obtain the signal's global degradation tendency and local random fluctuation components. Then, the global degradation trend and each fluctuation component are modeled using an echo state network (ESN) and a Bayesian optimized long short-term memory (LSTM) network, respectively. The final LIBs RUL prediction results are obtained by integrating the prediction outcomes. On three public LIBs datasets with distinct degradation characteristics, the performance of the proposed model is tested, and alternative prediction algorithms are compared. The results of the experiments show that the proposed model's maximum average absolute percentage error does not exceed 0.43%. The average relative error not more than 0.5%, indicating great accuracy and stability in prediction.

A Cerebellum-Inspired Prediction and Correction Model for Motion Control of a Musculoskeletal Robot

It is an important issue that how to regulate the existing control models of musculoskeletal robots to improve the ability of motion learning and generalization. In this paper, based on the motion modulation function of the cerebellum, a cerebellum-inspired prediction and correction model is proposed to carry out feedforward regulation of the original controller. Firstly, drawing on the reservoir computing mechanism of the cerebellar granular layer, the cerebellum prediction model is established by using the echo state network. Incremental learning for the network is achieved using the replay method, which is able to process control signals with different distributions. The cerebellum prediction network can accurately predict the motion results of the robot under the action of the time-series control signals. Secondly, referring to the neural pathways of the cerebellum, the cerebellum correction model is constructed. The network learning rules are designed by drawing on the long-term potentiation and depression processes of cerebellar synaptic plasticity. The characteristics of the parameters in the network weight update equations are further analyzed. And the hyperparameter update rules of the correction network weights are proposed. To simulate the function of the cerebellum involved in the limb ballistic movement, the adaptive adjustment method for cerebellum correction duration is proposed. The cerebellum correction network can accurately modulate the original control signals by using the motion prediction results. In experiments, a musculoskeletal robot is used to verify the movement effects under the control of the cerebellum model. The results show that the cerebellum-inspired model can effectively improve the motion accuracy and learning efficiency of the musculoskeletal robot, and enhance the motion generalization ability and system robustness of the robot.

Sustainability and robust decision-support strategy for multi-echelon supply chain system against disruptions

ABSTRACT Due to the higher nonlinearity and parametric sensitivity of multi-echelon supply chains, minor changes may result in a completely different evolution of supply chain capabilities. Realizing resilient and sustainable development goals for enterprise logistics requires great insights into management schemes in a volatile market. This paper deals with dynamic characteristics and resilience issues of managing multi-echelon supply chain systems against disruptions. Discrete-time nonlinear systems with time delay are extensively evaluated via computation of Lyapunov exponent, bifurcation diagram, permutation entropy, and Poincaré map. For time-series evolution, forecasting and predictability are assessed by the Hurst exponent. An echo state network (ESN) will be utilized to predict the future states of the supply chain dynamics. Based on a decision-support strategy with an ESN model, a resilience strategy concerning disturbing factors is implemented to ensure a sustainable supply chain network. Numerical simulations are demonstrated to validate dynamical behaviors and verify the efficiency of control theory with machine learning, guaranteeing the resilience and sustainability of a supply chain against market volatility. The test results demonstrate how machine learning algorithms can be integrated with management schemes. Finally, the presented methodology can ensure more supply chain visibility and traceability under highly uncertain markets.

Passive frustrated nanomagnet reservoir computing

Reservoir computing (RC) has received recent interest because reservoir weights do not need to be trained, enabling extremely low-resource consumption implementations, which could have a transformative impact on edge computing and in-situ learning where resources are severely constrained. Ideally, a natural hardware reservoir should be passive, minimal, expressive, and feasible; to date, proposed hardware reservoirs have had difficulty meeting all of these criteria. We, therefore, propose a reservoir that meets all of these criteria by leveraging the passive interactions of dipole-coupled, frustrated nanomagnets. The frustration significantly increases the number of stable reservoir states, enriching reservoir dynamics, and as such these frustrated nanomagnets fulfill all of the criteria for a natural hardware reservoir. We likewise propose a complete frustrated nanomagnet reservoir computing (NMRC) system with low-power complementary metal-oxide semiconductor (CMOS) circuitry to interface with the reservoir, and initial experimental results demonstrate the reservoir’s feasibility. The reservoir is verified with micromagnetic simulations on three separate tasks demonstrating expressivity. The proposed system is compared with a CMOS echo state network (ESN), demonstrating an overall resource decrease by a factor of over 10,000,000, demonstrating that because NMRC is naturally passive and minimal it has the potential to be extremely resource efficient.

Network traffic prediction model based on linear and nonlinear model combination

AbstractWe propose a network traffic prediction model based on linear and nonlinear model combination. Network traffic is modeled by an autoregressive moving average model, and the error between the measured and predicted network traffic values is obtained. Then, an echo state network is used to fit the prediction error with nonlinear components. In addition, an improved slime mold algorithm is proposed for reservoir parameter optimization of the echo state network, further improving the regression performance. The predictions of the linear (autoregressive moving average) and nonlinear (echo state network) models are added to obtain the final prediction. Compared with other prediction models, test results on two network traffic datasets from mobile and fixed networks show that the proposed prediction model has a smaller error and difference measures. In addition, the coefficient of determination and index of agreement is close to 1, indicating a better data fitting performance. Although the proposed prediction model has a slight increase in time complexity for training and prediction compared with some models, it shows practical applicability.

Estimating thermal performance of thermosyphons by artificial neural networks

The overheating of electronic devices has become very common with the advancement of technology, requiring the development of new alternatives for thermal control. Due to their excellent heat transfer ability, no external power is needed and they are adaptable to different geometries and applications. Thermosyphons are an excellent alternative for this thermal control. The thermal performance of thermosyphons is usually evaluated by their thermal resistance, and several variables were investigated to understand their influence on this parameter, such as working fluid, filling ratio, and slope. In that way, the fact that the thermal resistance depends on several variables makes its prediction complex and time-consuming. To overcome this issue, artificial intelligence-based methods, such as Artificial Neural Networks (ANNs), could be used. In this sense, an experimental investigation of the thermal performance of thermosyphon under different filling ratios (20 to 100%), slopes (45 and 90°), and heat loads (5 to 45 W) was made. The experimental data were then used as the database for different ANNs to predict the thermal resistance of a thermosyphon. For the experimental investigation, a thermosyphon of copper tube was built of 9.45 mm and 7.75 mm outer and inner diameter and a length of 500 mm. Its regions, evaporator, adiabatic section, and condenser, had 80, 20, and 100 mm, respectively. Distilled water was used as the working fluid. The evaporator was heated due to the Joule’s effect resulting from power dissipation in an electric ribbon wrapped in its length. The condenser was cooled with a 5 m/s air-forced convection. Regarding the use of ANNs, Unorganized Machines (UMs), composed of Extreme Learning Machines (ELM) and Echo State Networks (ESN), were proposed. As a means of comparison, the Radial Basis Function Network, and the Multilayer Perceptron (MLP), the most widely known neural architecture in the literature, were also applied. To estimate the thermal resistance of thermosyphon, the filling ratio, slope, and heat load were considered as inputs, and a total of 67 samples were used. Experimental results indicated that the best thermal performance occurs at a filling ratio of 40%, while the slope of 45° presented a better performance than 90°. The computational results revealed that the UMs could overcome the other methods, especially the ESN. The difference between the predicted and the experimental values was up to 25% for almost all cases. As a matter of reducing the experimental tests, applying ANN was essential.

Continual adaptation of federated reservoirs in pervasive environments

When performing learning tasks in pervasive environments, the main challenge arises from the need of combining federated and continual settings. The former comes from the massive distribution of devices with privacy-regulated data. The latter is required by the low resources of the participating devices, which may retain data for short periods of time. In this paper, we propose a setup for learning with Echo State Networks (ESNs) in pervasive environments. Our proposal focuses on the use of Intrinsic Plasticity (IP), a gradient-based method for adapting the reservoir’s non-linearity. First, we extend the objective function of IP to include the uncertainty arising from the distribution of the data over space and time. Then, we propose Federated Intrinsic Plasticity (FedIP), which is intended for client–server federated topologies with stationary data, and adapts the learning scheme provided by Federated Averaging (FedAvg) to include the learning rule of IP. Finally, we further extend this algorithm for learning to Federated Continual Intrinsic Plasticity (FedCLIP) to equip clients with CL strategies for dealing with continuous data streams. We evaluate our approach on an incremental setup built upon real-world datasets from human monitoring, where we tune the complexity of the scenario in terms of the distribution of the data over space and time. Results show that both our algorithms improve the representation capabilities and the performance of the ESN, while being robust to catastrophic forgetting.

Adaptive echo state network with a recursive inverse-free weight update algorithm

Echo State Network (ESN) is widely applied in sequence prediction, physics, and economics. Moore-Penrose inversion is a typical solving process of ESN. However, in the era of big data, the traditional inversion is computationally costly and the data may be irreversible. To solve this problem, a recursive inverse-free algorithm ESN model, called RIFESN, is introduced in this article. The model also introduces an adaptive activation function based on the error between target value and predicted value, which greatly improves the nonlinear mapping capability of the network. Three simulation experiments are conducted on the Mackey-Glass-17 time series, sunspots series and daily exchange rate summary of eight countries from 1990 to 2016, respectively. The results of the three simulations show that RIFESN has better prediction accuracy and operational speed, which proves that it has a good application prospect.

A Novel Echo State Network Autoencoder for Anomaly Detection in Industrial IoT Systems

The Industrial Internet of Things (IIoT) technology had a very strong impact on the realization of smart frameworks for detecting anomalous behaviors that could be potentially dangerous to a system. In this regard, most of the existing solutions involve the use of Artificial Intelligence (AI) models running on Edge devices, such as Intelligent Cyber Physical Systems (ICPS) typically equipped with sensing and actuating capabilities. However, the hardware restrictions of these devices make the implementation of an effective anomaly detection algorithm quite challenging. Considering an industrial scenario, where signals in the form of multivariate time-series should be analyzed to perform a diagnosis, Echo State Networks (ESNs) are a valid solution to bring the power of neural networks into low complexity models meeting the resource constraints. On the other hand, the use of such a technique has some limitations when applied in unsupervised contexts. In this paper, we propose a novel model that combines ESNs and autoencoders (ESN-AE) for the detection of anomalies in industrial systems. Unlike the ESN-AE models presented in the literature, our approach decouples the encoding and decoding steps and allows the optimization of both the processes while performing the dimensionality reduction. Experiments demonstrate that our solution outperforms other machine learning approaches and techniques we found in the literature resulting also in the best trade-off in terms of memory footprint and inference time.

Predicting aging transition using Echo state network.

It is generally known that in a mixture of coupled active and inactive nonlinear oscillators, the entire system may stop oscillating and become inactive if the fraction of active oscillators is reduced to a critical value. This emerging phenomenon, called the "aging transition," can be analytically predicted from the view point of cluster synchronization. One can question whether a model-free, data-driven framework based on neural networks could be used to foretell when such a system will cease oscillation. Here, we demonstrate how a straightforward ESN with trained output weights can accurately forecast both the temporal evaluation and the onset of collapse in coupled paradigmatic limit-cycle oscillators. In particular, we have demonstrated that an ESN can identify the critical fraction of inactive oscillators in a large all-to-all, small-world, and scale-free network when it is trained only with two nodes (one active and the other inactive) selected from three different pre-collapse regimes. We further demonstrate that ESN can anticipate aging transition of the network when trained with the mean-field dynamics of active and inactive oscillators.

Echo State Network With Probabilistic Regularization for Time Series Prediction

Echo State Network With Probabilistic Regularization for Time Series Prediction

A 65 nm 12.92-nJ/Inference Mixed-Signal Neuromorphic Processor for Image Classification

Spiking neural networks are a promising candidate for next-generation machine learning and are suitable for powerconstrained edge devices. In this paper, we present a mixed-signal neuromorphic processor that efficienctly implements an echo state network (ESN) and achieves high accuracy without a costly on-chip training process. The design employs a charge-domain computation circuit that efficiently realizes a leaky integrate and fire neuron. Combined with optimizing sparse connections, the proposed algorithm-hardware co-design approach results in a highly energy-efficient operation while delivering high accuracy. Fabricated in a 65nm LP process, the processor is measured to achieve 95.35% MNIST classification accuracy, which closely matches the software model, and energy efficiency of 12.92nJ per classification.

Neural Predictor-Based Dynamic Surface Parallel Control for MIMO Uncertain Nonlinear Strict-Feedback Systems

We propose a neural predictor-based dynamic surface parallel control method for a class of uncertain nonlinear systems in this brief. The dynamics of the physical system is in the strict-feedback form and subject to multi-input, multi-output, and uncertain nonlinearities. The parallel control method is developed based on an ACP methodology, including three steps. Firstly, an artificial system to the physical system is developed using an echo state network-based neural predictor structure. Next, a high-order tuner-based computational experiment is developed to achieve online adaptive training of the echo state network. Finally, parallel execution is developed by using a second-order linear tracking differentiator-based dynamic surface control approach. The total closed-loop system can be proved to be input-to-state stable. The effectiveness of the proposed theoretical results is demonstrated by a simulation of trajectory tracking of an autonomous surface vehicle.

Rolling decomposition method in fusion with echo state network for wind speed forecasting

Accurate wind speed forecasting is beneficial to ensure the safe and stable operation of power systems, improve economic benefits, and promote the healthy development of the wind power industry. This study develops a novel hybrid model called VMD-ESN-STO combining the rolling variational mode decomposition (VMD), echo state network (ESN), and subseries to original series (STO) structure for wind speed forecasting. In this model, the rolling method is not only used in decomposition, but also in training and forecasting. The rolling VMD is used to decompose the original wind speed series into several subseries according to the rolling schema, ESN is used to forecast, and STO structure determines the input and output of the forecasting model. Four wind speed datasets are utilized for wind speed forecasting experiments to validate the applicability and accuracy of the developed model. Mean absolute percentage errors of VMD-ESN-STO in the four datasets are 4.4511%, 2.4451%, 4.1400%, and 2.6178%, respectively, which are far less than the errors for the six comparative models. The developed VMD-ESN-STO is an appropriate tool for wind speed forecasting due to its superior forecasting performance.

Using normalized echo state network to detect abnormal ECG patterns

AbstractCardiovascular disease (CVD) stands as a prominent contributor to human mortality. Electrocardiogram (ECG) represents a widely adopted noninvasive method employed by clinicians to detect and diagnose CVDs. Nonetheless, conventional ECG‐based detection approaches for cardiac disorders tend to be time‐consuming and inefficient, necessitating the need for more effective solutions. Recent studies have highlighted the effectiveness of the echo state network (ESN) in detecting abnormal ECG patterns. However, traditional ESN models often face challenges such as unstable training and convergence difficulties due to variations in the range of reservoir state values. To address this issue, this study introduces a novel approach called the normalized echo state network (NESN). The NESN method normalizes the states of all neurons within the reservoir before applying the nonlinear activation function. In our study, we conducted performance evaluations of the proposed model using the MIT‐BIH arrhythmia database. We performed a synergistical analysis to investigate the impact of reservoir parameters on the network performance. The experimental results demonstrated promising outcomes, with an accuracy of 99.1% and an F1‐score of 96.4%. Specifically, for detecting abnormal ECG patterns, our model achieved a sensitivity of 90.2%, a positive predictive value of 96.6%, and a specificity of 99.8%. These results highlight the superior performance of our classifier compared to most traditional mainstream heartbeat detection methods and ring topology ESN model.

When Deep is not Enough: Towards Understanding Shallow and Continual Learning Models in Realistic Environmental Sound Classification for Robots

Although deep learning models are state-of-the-art models in audio classification, they fall short when applied in developmental robotic settings and human–robot interaction (HRI). The major drawback is that deep learning relies on supervised training with a large amount of data and annotations. In contrast, developmental learning strategies in human–robot interaction often deal with small-scale data acquired from HRI experiments and require the incremental addition of novel classes. Alternatively, shallow learning architectures that enable fast and yet robust learning are provided by simple distance metric-based learning and neural architectures implementing the reservoir computing paradigm. Similarly, continual learning algorithms receive more attention in the last years as they can integrate stable perceptual feature extraction using pre-trained deep learning models with open-set classification. As our research centers around reenacting the incremental learning of audio cues, we conducted a study on environmental sound classification using the iCaRL as well as the GDumb continual learning algorithms in comparison with a popular classifier in this domain, the kNN classifier, as well as employing an Echo State Network. We contrast our results with those obtained from a VGGish network that serves here as the performance upper bound that allows us to quantify the performance differences and to discuss current issues with continual learning in the audio domain. As only little is known about using shallow models or continual learning in the audio domain, we pass on additional techniques like data augmentation and create a simple experimental pipeline that is easy to reproduce. Although our selected algorithms are partially inferior in performance compared to the upper bound, our evaluation on three environmental sound datasets shows promising performance using continual learning for a subset of the DCASE2019 challenge dataset and the ESC10 dataset. As we do not address benchmarking in this paper, our study provides a good foundation for further research and computational improvements on shallow and continual learning models for robotic applications in the audio domain.

Generalised Predictive Control of Nonlinear Systems Using the Echo State Network and the Nelder-Mead Method

Generalised Predictive Control of Nonlinear Systems Using the Echo State Network and the Nelder-Mead Method