Echo State Neural Network Research Articles

Echo state network implementation for chaotic time series prediction

The implementation of an Echo State Neural Network (ESNN) for chaotic time series prediction is introduced. First, the ESNN is simulated using floating-point arithmetic and afterwards fixed-point arithmetic. The synthesis of the ESNN is done in a field-programmable gate array (FPGA), in which the activation function of the neurons’ outputs is a hyperbolic tangent one, and is approximated with a new design of quadratic order b-splines and four integer multipliers. The FPGA implementation of the ESNN is applied to predict four chaotic time series associated to the Lorenz, Chua, Lü, and Rossler chaotic oscillators. The experimental results show that with 50 hidden neurons, the fixed-point arithmetic is good enough when using 15 or 16 bits in the fractional part: using more bits does not reduce the mean-squared error prediction. The neurons are limited to four inputs in the hidden layer to achieve a more efficient hardware implementation, guaranteeing a prediction of more than 10 steps ahead.

Echo State Neural Network Based on an Improved Gray Wolf Algorithm Predicts Porosity through Logging Data.

In oil exploration and development, many reservoir parameters are very essential for reservoir description, especially porosity. The porosity obtained by indoor experiments is reliable, but human and material resources will be greatly invested. Experts have introduced machine learning into the field of porosity prediction but with the shortcomings of traditional machine learning models, such as hyperparameter abuse and poor network structure. In this paper, a meta-heuristic algorithm (Gray Wolf Optimization algorithm) is introduced to optimize the ESN (echo state neural) network for logging porosity prediction. Tent mapping, a nonlinear control parameter strategy, and PSO (particle swarm optimization) thought are introduced to optimize the Gray Wolf Optimization algorithm to improve the global search accuracy and avoid local optimal solutions. The database is constructed by using logging data and porosity values measured in the laboratory. Five logging curves are used as model input parameters, and porosity is used as the model output parameter. At the same time, three other prediction models (BP neural network, least squares support vector machine, and linear regression) are introduced to compare with the optimized models. The research results show that the improved Gray Wolf Optimization algorithm has more advantages than the ordinary Gray Wolf Optimization algorithm in terms of super parameter adjustment. The IGWO-ESN neural network is better than all machine learning models mentioned in this paper (GWO-ESN, ESN, BP neural network, least squares support vector machine, and linear regression) in terms of porosity prediction accuracy.

Robust fault diagnosis of a high-voltage circuit breaker via an ensemble echo state network with evidence fusion

Reliable mechanical fault diagnosis of high-voltage circuit breakers is important to ensure the safety of electric power systems. Recent fault diagnosis approaches are mostly based on a single classifier whose performance relies heavily on expert prior knowledge. In this study, we propose an improved Dempster–Shafer evidence theory fused echo state neural network, an ensemble classifier for fault diagnosis. Evidence credibility is calculated through the evidence deviation matrix and the segmented circle function and employed as credibility weights to rectify the raw evidence. Then, an improved Dempster–Shafer evidence fusion algorithm is proposed to fuse evidence from different echo state network modules and sensors. Unlike conventional classifiers, the proposed methodology consists of multiple echo state neural network modules. It has better flexibility and stronger robustness, and its model performance is not sensitive to network parameters. Comparative analysis indicates that it can handle the paradox evidence fusion analysis and thus can achieve better diagnostic performance. The superiority of the reported fault diagnosis approaches is verified with the experimental data of a ZN12 high-voltage circuit breaker.

Predicting Stellar Mass Accretion: An Optimized Echo State Network Approach in Time Series Modeling

Modeling the dynamics of the formation and evolution of protostellar disks as well as the history of stellar mass accretion typically involve the numerical solution of complex systems of coupled differential equations. The resulting mass accretion history of protostars is known to be highly episodic due to recurrent instabilities and also exhibits short timescale flickering. By leveraging the strong predictive abilities of neural networks, we extract some of the critical temporal dynamics experienced during the mass accretion including periods of instability. Particularly, we utilize a novel form of the Echo-State Neural Network (ESN), which has been shown to efficiently deal with data having inherent nonlinearity. We introduce the use of Optimized-ESN (Opt-ESN) to make model-independent time series forecasting of mass accretion rate in the evolution of protostellar disks. We apply the network to multiple hydrodynamic simulations with different initial conditions and exhibiting a variety of temporal dynamics to demonstrate the predictability of the Opt-ESN model. The model is trained on simulation data of $\sim 1-2$ Myr, and achieves predictions with a low normalized mean square error ($\sim 10^{-5}$ to $10^{-3}$) for forecasts ranging between 100 and 3800 yr. This result shows the promise of the application of machine learning based models to time-domain astronomy.

Event-trigger-based cluster coordinated control of spacecraft swarm under switching topology

This paper investigates the cluster coordination problem of spacecraft swarms with non-negative defined directed communication topology in the context of multi-mission parallel operations. In this design, an intrinsic plasticity based echo state network is established to approximate the nonlinear term of relative attitude dynamics, and an improved event-triggered transmission mechanism is proposed to decrease the redundant communication among spacecraft. A novel cluster coordination control law under fixed non-negative topology is developed based on the echo state neural network and the event-triggered transmission mechanism. Furthermore, an extended cluster coordinated control law is proposed for the switching topology case. By using Lyapunov stability theory and graph theory, it is demonstrated that intra-cluster attitude coordination can be achieved under the proposed control laws and Zeno behavior can be excluded. Finally, numerical examples are given to illustrate the validity and superiority of our results.

Noise Impact on a Recurrent Neural Network with a Linear Activation Function

In this paper we analyze an echo state neural network (ESN) in the presence of uncorrelated additive and multiplicative white Gaussian noise. Here we consider the case where artificial neurons have a linear activation function with different slope coefficients. We consider the influence of the input signal, memory and connection matrices on the accumulation of noise. We have found that the general view of variance and the signal-to-noise ratio of the ESN output signal is similar to only one neuron. The noise is less accumulated in ESN with a diagonal reservoir connection matrix with a large “blurring” coefficient. This is especially true of uncorrelated multiplicative noise.

Experimental evidence that rill-bed morphology is governed by emergent nonlinear spatial dynamics

Past experimental work found that rill erosion occurs mainly during rill formation in response to feedback between rill-flow hydraulics and rill-bed roughness, and that this feedback mechanism shapes rill beds into a succession of step-pool units that self-regulates sediment transport capacity of established rills. The search for clear regularities in the spatial distribution of step-pool units has been stymied by experimental rill-bed profiles exhibiting irregular fluctuating patterns of qualitative behavior. We hypothesized that the succession of step-pool units is governed by nonlinear-deterministic dynamics, which would explain observed irregular fluctuations. We tested this hypothesis with nonlinear time series analysis to reverse-engineer (reconstruct) state-space dynamics from fifteen experimental rill-bed profiles analyzed in previous work. Our results support this hypothesis for rill-bed profiles generated both in a controlled lab (flume) setting and in an in-situ hillside setting. The results provide experimental evidence that rill morphology is shaped endogenously by internal nonlinear hydrologic and soil processes rather than stochastically forced; and set a benchmark guiding specification and testing of new theoretical framings of rill-bed roughness in soil-erosion modeling. Finally, we applied echo state neural network machine learning to simulate reconstructed rill-bed dynamics so that morphological development could be forecasted out-of-sample.

A comparative study of the performance of methods for f-wave extraction

Objective. This study proposes a novel technique for atrial fibrillatory waves (f-waves) extraction and investigates the performance of the proposed method comparing with different f-wave extraction methods. Approach. We propose a novel technique combining a periodic component analysis (PiCA) and echo state network (ESN) for f-waves extraction, denoted PiCA-ESN. PiCA-ESN benefits from the advantages of using both source separation and nonlinear adaptive filtering. PiCA-ESN is evaluated by comparing with other state-of-the-art approaches, which include template subtraction technique based on principal component analysis, spatiotemporal cancellation, nonlinear adaptive filtering using an echo state neural network, and a source separation technique based on PiCA. Quality assessment is performed on a recently published reference database including a large number of simulated ECG signals in atrial fibrillation (AF). The performance of the f-wave extraction methods is evaluated in terms of signal quality metrics (SNR, ΔSNR) and robustness of f-wave features. Main results. The proposed method offers the best signal quality performance, with a ΔSNR of approximately 22 dB across all 8 sets of the reference database, as well as the most robust extraction of f-wave features, with 75% of all estimates of dominant atrial frequency well below 1 Hz.

Research on water quality prediction model based on echo state network

Using artificial neural network (ANN) to solve the problem of time series water quality prediction has become increasingly mature. In this paper, through the study of leaky-integral echo state neural network (Leaky ESN), combined with the historical water quality data of Dongzhen Reservoir in Fujian Province, a single-day water quality prediction model was constructed, and the Bayesian optimization algorithm was used to realize the automatic optimization of hyper-parameters in the network. On this basis, multi-day prediction models were constructed by further improving the network, which used the historical water quality data of the previous 7 days to predict the water quality of the next 3 days, 5 days and 7 days. Then the prediction models were applied to the water quality prediction of the study. The experimental results show that the single-day prediction model with Bayesian optimization has high accuracy. The multi-day prediction models can also achieve good prediction effect, and have more practical application value. They are more suitable for early warning of water quality.

TSK fuzzy echo state neural network: a hybrid structure for black-box nonlinear systems identification

TSK fuzzy echo state neural network: a hybrid structure for black-box nonlinear systems identification

Adaptive neural control of PEMFC system based on data-driven and reinforcement learning approaches

Proton exchange membrane fuel cell systems are being increasingly put forward as hydrogen energy carrier converters. Recent advancements in reliability strategies have been stimulated through maintaining a healthy operating condition of the system and covering plant faults. However, it is observed that occurrence or even the mitigation of these faults cause multilateral effects that can potentially destabilize the normal operation of the system. In the active fault tolerant control strategy, two modules are designed to fault management. The diagnostic module identifies the apparent fault and identifies the corrective commands, then the re-design module adapts the controller to dynamic system changes. In order to improve the generic characteristics of the re-design module, this paper presents a data-driven neural controller capable to automatically adapt to system health states. The developed approach comes from the machine learning class and combines adaptive dynamic programming, deep echo-state neural network models and fuzzy logic learning. The proposed controller is evaluated under occurrence of channels flooding and membrane drying faults, but also actuators and water purging disturbances. Simulation and experimental results show the effectiveness of the proposed data-driven approach without prior neural model training, while guaranteeing the stability and learning convergence of the adaptive controller.

Classifying Unimanual and Bimanual Upper Extremity Tasks in Individuals Post-Stroke.

After stroke, many individuals develop impairments that lead to compensatory motions. Compensation allows individuals to achieve tasks but has long-term detrimental effects and represents maladaptive motor strategies. Increased use of bimanual motions may serve as a biomarker for recovery (and the reduction of reliance on compensatory motion), and tracking such motion using sensor data may provide critical data for health care specialists. However, past work by the authors demonstrated individual variation in motor strategies results in noisy and chaotic sensor data. The goal of the current work is to develop classifiers capable of differentiating unimanual, bimanaual asymmetric, and bimanual symmetric gestures using wearable sensor data. Twenty participants post-stroke (and 20 age-matched controls) performed a set of tasks under the supervision of a trained occupational therapist. Sensor data were recorded for each task. Classifiers were developed using artificial neural networks (ANNs) as a baseline, and the echo state neural network (ESNN) which has demonstrated efficacy with chaotic data. We find that, for control and post-stroke participants, the ESNN results in improved testing accuracy performance (91.3% and 80.3%, respectively). These results suggest a novel method for classifying gestures in individuals post-stroke, and the developed classifiers may facilitate longitudinal monitoring and correction of compensatory motion.

Direct adaptive control for nonlinear systems using a TSK fuzzy echo state network based on fractional-order learning algorithm

This paper presents a new Takagi-Sugeno-Kang fuzzy Echo State Neural Network (TSKFESN) structure to design a direct adaptive control for uncertain SISO nonlinear systems. The proposed TSKFESN structure is based on the echo state neural network framework containing multiple sub-reservoirs. Each sub-reservoir is weighted with a TSK fuzzy rule. The adaptive law of the TSKFESN-based direct adaptive controller is derived by using a fractional-order sliding mode learning algorithm. Moreover, the Lyapunov stability criterion is employed to verify the convergence of the fractional-order adaptive law of the controller parameters. The evaluation of the proposed direct adaptive control scheme is verified using two case studies, the regulation problem of a torsional pendulum and the speed control of a direct current (DC) machine as a real-time application. The simulation and the experimental results show the effectiveness of the proposed control scheme.

Novel hybrid model based on echo state neural network applied to the prediction of stock price return volatility

The prediction of stock price return volatilities is important for financial companies and investors to help to measure and managing market risk and to support financial decision-making. The literature points out alternative prediction models - such as the widely used heterogeneous autoregressive (HAR) specification - which attempt to forecast realized volatilities accurately. However, recent variants of artificial neural networks, such as the echo state network (ESN), which is a recurrent neural network based on the reservoir computing paradigm, have the potential for improving time series prediction. This paper proposes a novel hybrid model that combines HAR specification, the ESN, and the particle swarm optimization (PSO) metaheuristic, named HAR-PSO-ESN, which combines the feature design of the HAR model with the prediction power of ESN, and the consistent PSO metaheuristic approach for hyperparameters tuning. The proposed model is benchmarked against existing specifications, such as autoregressive integrated moving average (ARIMA), HAR, multilayer perceptron (MLP), and ESN, in forecasting daily realized volatilities of three Nasdaq (National Association of Securities Dealers Automated Quotations) stocks, considering 1-day, 5-days, and 21-days ahead forecasting horizons. The predictions are evaluated in terms of r-squared and mean squared error performance metrics, and the statistical comparison is made through a Friedman test followed by a post-hoc Nemenyi test. Results show that the proposed HAR-PSO-ESN hybrid model produces more accurate predictions on most of the cases, with an average R2 (coefficient of determination) of 0.635, 0.510, and 0.298, an average mean squared error of 5.78 × 10−8, 5.78 × 10−8, and 1.16 × 10−7, for 1, 5, and 21 days ahead on the test set, respectively. The improvement is statistically significant with an average rank of 1.44 considering the three different datasets and forecasting horizons.

Implantation modified deep echo state neural networks and improve harmony clustering algorithm for optimal and energy efficient path in mobile sink

Wireless network sensors based on the mobile sink are regarded to be a common network and used in various fields in the last few years, they are thought to be easy to use, but contain the problem of energy loss and are affected by an energy hole problem, as it depends on batteries. This paper proposes a solution to this problem by using an innovative objective function for a consistent distributing of cluster heads, the enhanced harmony search based routing protocols based on energy equilibrated node clustering protocol. In order to route the data packet among the sink and cluster heads, an enhanced modified deep echo state neural network is suggested. The efficiency of a projected integrated clustering and routing protocol has been investigated at 500 nodes, and the 96 per cent success data for the proposed algorithm is given using the average energy consumption, send and receive packaged and optimum numbers of CH.

Implantation modified deep echo state neural networks and improve harmony clustering algorithm for optimal and energy efficient path in mobile sink

Wireless network sensors based on the mobile sink are regarded to be a common network and used in various fields in the last few years, they are thought to be easy to use, but contain the problem of energy loss and are affected by an energy hole problem, as it depends on batteries. This paper proposes a solution to this problem by using an innovative objective function for a consistent distributing of cluster heads, the enhanced harmony search based routing protocols based on energy equilibrated node clustering protocol. In order to route the data packet among the sink and cluster heads, an enhanced modified deep echo state neural network is suggested. The efficiency of a projected integrated clustering and routing protocol has been investigated at 500 nodes, and the 96 per cent success data for the proposed algorithm is given using the average energy consumption, send and receive packaged and optimum numbers of CH.

Prediction Method of Ocean Wave Spectrum Based on an Echo State Network

Zhang, X.; Yang, B.; Sun, H., and Zhang, S., 2020. Prediction method of ocean wave spectrum based on an echo state network. In: Zheng, C.W.; Wang, Q.; Zhan, C., and Yang, S.B. (eds.), Air-Sea Interaction and Coastal Environments of the Maritime and Polar Silk Roads. Journal of Coastal Research, Special Issue No. 99, pp. 319–323. Coconut Creek (Florida), ISSN 0749-0208.In order to realize the prediction of ocean wave spectrum under strong nonlinear conditions, a new method combining the empirical mode decomposition (EMD) and an echo state neural network (ESN) is proposed based on ocean wave spectral data measured by shipborne X-band radar. The proposed method uses empirical mode decomposition to decompose the time series of the strongly nonlinear wavelet spectral density values so that the nonlinearity and nonstationarity of the obtained subsequence is greatly reduced compared with the original sequence, and the echo state neural network can be applied separately. By superimposing the prediction results, the predicted values of the wavelet spectral density can be obtained, and the entire ocean wave spectral information can be further combined. The results show that the method can effectively solve the problem of poor prediction effects under strong nonlinear conditions. The method can provide a certain basis for ships to obtain wave spectral information in real time and improve the practicability of shipborne wave radar systems.

Degradation prediction of PEM fuel cell based on artificial intelligence

In the last years, Proton Exchange Membrane Fuel Cells (PEMFC) became a promising energy converter for both transportation and stationary applications. However, durability of fuel cells still needs to be improved to achieve a widespread deployment. Degradation mechanisms and aging laws are not yet fully understood. Therefore, long-term durability tests are necessary to get more information. Moreover, degradation models are requested to estimate the remaining useful life of the system and take adequate corrective actions to optimize durability and availability. This paper presents in a first part the results of a long-term durability test performed on an open cathode fuel cell system operated during 5000 h under specific operating conditions including start/stop and variable ambient temperature. Performance evolution and degradation mechanisms are then analyzed to understand influence of operating conditions and how to extend the durability. In a second part of the paper, the results are used to build a degradation model based on echo state neural network in order to predict the performance evolution. Results of the degradation prediction are very promising as the normalized root mean square error remains very low with a prediction time over 2000 h.

Adaptive Neural Funnel Control for Nonlinear Two-Inertia Servo Mechanisms With Backlash

This paper proposes a modified adaptive neural control algorithm with a funnel function for nonlinear two-inertia servomechanisms with backlash and external disturbance. A continuous tracking differentiator is employed to replace the first-order filter in the traditional dynamic surface control design. A novel funnel function is employed and used in control design to improve the control performance. Then, an adaptive neural dynamic surface controller based on funnel control is presented to guarantee the tracking performance of two-inertia servomechanisms. In addition, the unknown dynamics, including backlash, nonlinear friction, and external disturbance, are approximated by using the echo state neural network and compensated online. The comparative experimental results validate the effectiveness of the developed control algorithm based on a two-inertia servo mechanism.

Know Your Body Through Intrinsic Goals.

The first “object” that newborn children play with is their own body. This activity allows them to autonomously form a sensorimotor map of their own body and a repertoire of actions supporting future cognitive and motor development. Here we propose the theoretical hypothesis, operationalized as a computational model, that this acquisition of body knowledge is not guided by random motor-babbling, but rather by autonomously generated goals formed on the basis of intrinsic motivations. Motor exploration leads the agent to discover and form representations of the possible sensory events it can cause with its own actions. When the agent realizes the possibility of improving the competence to re-activate those representations, it is intrinsically motivated to select and pursue them as goals. The model is based on four components: (1) a self-organizing neural network, modulated by competence-based intrinsic motivations, that acquires abstract representations of experienced sensory (touch) changes; (2) a selector that selects the goal to pursue, and the motor resources to train to pursue it, on the basis of competence improvement; (3) an echo-state neural network that controls and learns, through goal-accomplishment and competence, the agent's motor skills; (4) a predictor of the accomplishment of the selected goals generating the competence-based intrinsic motivation signals. The model is tested as the controller of a simulated simple planar robot composed of a torso and two kinematic 3-DoF 2D arms. The robot explores its body covered by touch sensors by moving its arms. The results, which might be used to guide future empirical experiments, show how the system converges to goals and motor skills allowing it to touch the different parts of own body and how the morphology of the body affects the formed goals. The convergence is strongly dependent on competence-based intrinsic motivations affecting not only skill learning and the selection of formed goals, but also the formation of the goal representations themselves.