Conventional Echo State Network Research Articles

Enhancing accuracy and interpretability in EEG-based medical decision making using an explainable ensemble learning framework application for stroke prediction

Medical decision making increasingly relies on machine learning algorithms to analyze complex patient data and provide recommendations. However, the lack of interpretability in “black box” models has limited their adoption in clinical practice, which demands transparency and justification. Echo State Networks (ESNs), a recurrent neural network architecture, have shown promise for medical Decision Support Systems (DSS) applications due to their accuracy in modeling time series data, such as physiological signals. However, conventional ESNs suffer from a lack of interpretability, which limits their usefulness for clinical decisions. This study proposes a multi-level framework to optimize both accuracy and interpretability for improved medical decision support. The framework is specifically designed for EEG data, which represents a non-invasive and continuous recording of brain activity. EEG data is ideal for recording information about the internal state of the brain, which is not always translated by perceptible external manifestations. The framework consists of four components: (1) data preprocessing and optimized feature selection utilizing a filter-based feature importance algorithm, (2) ensemble learning via Ensemble ESNs (E-ESNs) that combines the predictions of multiple ESNs to reduce variance and improve accuracy, (3) Global and Local model explanation techniques to allow users to understand the decision-making process and identify the most influential features for each prediction, and (4) decision-making for presenting diagnoses and recommendations. The framework was evaluated on an EEG dataset for stroke prediction, a valuable use case for informed clinical decisions and resource allocation. The results showed that the framework significantly outperformed baseline related works with an accuracy of 96.5% and provides insights into the E-ESN model's predictions. This transparency enhances trust and comprehension among healthcare practitioners. These findings suggest that the proposed framework can be a valuable tool for medical DSS applications.

Novel Diversified Echo State Network for Improved Accuracy and Explainability of EEG-based Stroke Prediction

Echo State Networks (ESNs) are a powerful machine learning technique that can be used for EEG-based stroke prediction. However, conventional ESNs suffer from two main limitations: they are not always accurate, and they are not always interpretable. This paper presents a novel multi-level framework that addresses these limitations. The framework consists of three main components: optimized feature extraction, ensemble learning, and output refinement for improved interpretability. The optimized feature extraction component uses a novel algorithm to extract features from EEG data that are more relevant to stroke prediction. The ensemble learning component uses a diversified Echo State Networks (D-ESN) to combine the predictions of multiple ESNs, which improves the accuracy of the predictions. The output improvement component uses two Explainability techniques, LIME and ELI5, to gain insight into the decision-making of the D-ESN model. These techniques allow users to see how each feature in the dataset contributed to the model's prediction. The framework was evaluated on a well-known EEG dataset from stroke patients. The experimental results showed that the framework significantly outperformed baseline approaches in terms of both accuracy with 95% and interpretability. These results suggest that the proposed framework has the potential to advance the field of stroke prediction and enable informed decision-making in clinical settings.

An octonion-based nonlinear echo state network for speech emotion recognition in Metaverse

While the Metaverse is becoming a popular trend and drawing much attention from academia, society, and businesses, processing cores used in its infrastructures need to be improved, particularly in terms of signal processing and pattern recognition. Accordingly, the speech emotion recognition (SER) method plays a crucial role in creating the Metaverse platforms more usable​ and enjoyable for its users. However, existing SER methods continue to be plagued by two significant problems in the online environment. The shortage of adequate engagement and customization between avatars and users is recognized as the first issue and the second problem is related to the complexity of SER problems in the Metaverse as we face people and their digital twins or avatars. This is why developing efficient machine learning (ML) techniques specified for hypercomplex signal processing is essential to enhance the impressiveness and tangibility of the Metaverse platforms. As a solution, echo state networks (ESNs), which are an ML powerful tool for SER, can be an appropriate technique to enhance the Metaverse’s foundations in this area. Nevertheless, ESNs have some technical issues restricting them from a precise and reliable analysis, especially in the aspect of high-dimensional data. The most significant limitation of these networks is the high memory consumption caused by their reservoir structure in face of high-dimensional signals. To solve all problems associated with ESNs and their application in the Metaverse, we have come up with a novel structure for ESNs empowered by octonion algebra called NO2GESNet. Octonion numbers have eight dimensions, compactly display high-dimensional data, and improve the network precision and performance in comparison to conventional ESNs. The proposed network also solves the weaknesses of the ESNs in the presentation of the higher-order statistics to the output layer by equipping it with a multidimensional bilinear filter. Three comprehensive scenarios to use the proposed network in the Metaverse have been designed and analyzed, not only do they show the accuracy and performance of the proposed approach, but also the ways how SER can be employed in the Metaverse platforms.

Weighted error-output recurrent echo kernel state network for multi-step water level prediction

With development of information techniques in navigation and shipping, machine learning algorithms are applied in enhancing navigation safety. One of critical areas, which attracts lots of attention from scientists and researchers, is water level prediction. Although randomization-based algorithms obtain good performance in water level prediction, these algorithms still have their own limitations. For example, unstable prediction problem caused by random weights selection, and the error accumulation problem caused by the conventional recurrent algorithm. In this study, we combine three proposed approach with the conventional echo state network. Firstly, the Gaussian kernel method transforms the input features into a high-dimensional features, which in some extent improves the forecasting accuracy. Secondly, kernel reservoir states are proposed. It not only abandons the random selected weights, but it also makes hidden neurons to connect closely. Lastly, a novel weighted error-output recurrent multi-step algorithm is proposed. It uses previous forecasting errors to update the current output weights in order to overcome the error accumulation problem. Based on above three approaches, a multi-step water level prediction model is proposed called Weighted Error-output Recurrent Echo Kernel State Network (WER-EKSN). The experimental results and statistical analysis represent that our proposed model has better forecasting performance than other compared models. It not only has the superior ability in water level prediction, but it also provides the evidences for the management of transportation in water-land, such as flood protection, and management of ship route.

Integer Echo State Networks: Efficient Reservoir Computing for Digital Hardware.

We propose an approximation of echo state networks (ESNs) that can be efficiently implemented on digital hardware based on the mathematics of hyperdimensional computing. The reservoir of the proposed integer ESN (intESN) is a vector containing only n -bits integers (where is normally sufficient for a satisfactory performance). The recurrent matrix multiplication is replaced with an efficient cyclic shift operation. The proposed intESN approach is verified with typical tasks in reservoir computing: memorizing of a sequence of inputs, classifying time series, and learning dynamic processes. Such architecture results in dramatic improvements in memory footprint and computational efficiency, with minimal performance loss. The experiments on a field-programmable gate array confirm that the proposed intESN approach is much more energy efficient than the conventional ESN.

On the effectiveness of Gated Echo State Networks for data exhibiting long-term dependencies

In the context of recurrent neural networks, gated architectures such as the GRU have contributed to the development of highly accurate machine learning models that can tackle long-term dependencies in the data. However, the training of such networks is performed by the expensive algorithm of gradient descent with backpropagation through time. On the other hand, reservoir computing approaches such as Echo State Networks (ESNs) can produce models that can be trained efficiently thanks to the use of fixed random parameters, but are not ideal for dealing with data presenting long-term dependencies. We explore the problem of employing gated architectures in ESNs from both theoretical and empirical perspectives. We do so by deriving and evaluating a necessary condition for the non-contractivity of the state transition function, which is important to overcome the fading-memory characterization of conventional ESNs. We find that using pure reservoir computing methodologies is not sufficient for effective gating mechanisms, while instead training even only the gates is highly effective in terms of predictive accuracy.

Physics-informed echo state networks

We propose a physics-informed echo state network (ESN) to predict the evolution of chaotic systems. Compared to conventional ESNs, the physics-informed ESNs are trained to solve supervised learning tasks while ensuring that their predictions do not violate physical laws. This is achieved by introducing an additional loss function during the training, which is based on the system's governing equations. The additional loss function penalizes non-physical predictions without the need of any additional training data. This approach is demonstrated on a chaotic Lorenz system and a truncation of the Charney–DeVore system. Compared to the conventional ESNs, the physics-informed ESNs improve the predictability horizon by about two Lyapunov times. This approach is also shown to be robust with regard to noise. The proposed framework shows the potential of using machine learning combined with prior physical knowledge to improve the time-accurate prediction of chaotic dynamical systems.

Novel soft sensor development using echo state network integrated with singular value decomposition: Application to complex chemical processes

It is of great importance to develop advanced soft sensors for ensuring the safety and stability of complex industrial processes. Unluckily, with the increasing scale of chemical processes, it becomes more and more demanding to develop soft sensor with high accuracy. In addition, most of industrial processes are dynamic. As a result, the soft sensors developed using static models cannot achieve acceptable performance. In order to handle this problem, the Echo state network (ESN) as a kind of recurrent neural network is selected. However, the output weights of ESN are calculated linearly. On one hand, the collinear in the reserve layer outputs may decrease the performance; on the other hand, the over-fitting problem may occur. To enhance and improve the ESN performance, singular value decomposition based ESN (SVD-ESN) is presented. In the SVD-ESN method, the singular value decomposition instead of the traditional least square is adopted to calculate the weights between the output layer and the reserve layer. Through singular value analysis in the outputs of the reserve layer, appropriate defining parameters are selected to enhance the accuracy and ensure the computing speed. As a result, the collinearity and over-fitting problem is solved; then the performance of ESN is enhanced. To test and validate the performance of SVD-ESN, the proposed SVD-ESN is developed as soft sensor for the High Density Polyethylene (HDPE) production process and Purified Terephthalic Acid (PTA) production process. Compared with the conventional ESN, Extreme Learning Machine (ELM), Dynamic Window based ELM (DW-ELM) and Long Short-Term Memory (LSTM), the simulation results show that the proposed SVD-ESN model obtains better performance in terms of prediction accuracy, which conforms that the proposed SVD-ESN can be used as an effective dynamic model for developing accurate soft sensors.

Optimizing simple deterministically constructed cycle reservoir network with a Redundant Unit Pruning Auto-Encoder algorithm

Echo State Network (ESN) is a specific form of recurrent neural network, which displays very rich dynamics owing to its reservoir based hidden neurons. In the issue, ESN is viewed as a powerful approach to model real-valued time series processes. Nevertheless, ESN has been criticized for its manually experienced or brute-force searching parameters, such as initial input weights and reservoir layer weights, i.e., the conventional randomly generated ESN is unlikely to be optimal because the reservoir layer weights and input layer weights are created randomly. Simple Cycle Reservoir Network (SCRN), which constitutes a type of conclusively constructed input and internal layer weights, can yield performance comparable with conventional ESN. A Redundant Unit Pruning Auto-Encoder (RUP-AE) algorithm is proposed to optimize the input layer weights of SCRN and for resolving the dilemma of ill-conditioned output weights matrix in SCRN, through an unsupervised pre-training process. Initially, the output weights matrix of SCRN is pre-trained by pseudo-inverse algorithm through training data. Then, the pre-trained output weights matrix is pruned by a Redundant Unit Pruning (RUP) algorithm. Finally, the pruned output weights matrix of SCRN is injected to the input weights matrix to ensure the specificity of the auto-encoder. Three tasks, namely nonlinear time series system identification task, real-valued time series benchmark, and standard chaotic time series benchmark, are applied to demonstrate the advantage and superiority of RUP-AE. Extensive experimental results show that our RUP-AE is effective in improving the performance of SCRN. Meanwhile, RUP-AE is able to resolve the dilemma of ill-conditioned output weights matrix in SCRN.

Time Series Prediction Model of Grey Wolf Optimized Echo State Network

As a novel recursion neural network, Echo State Networks (ESN) are characterized by strong nonlinear prediction capability and effective and straightforward training algorithms. However, conventional ESN predictions require a large volume of training samples. Meanwhile, the time sequence data are complicated and unstable, resulting in insufficient learning of this network and difficult training. As a result, the accuracies of conventional ESN predictions are limited. Aimed at this issue, a time series prediction model of Grey Wolf optimized ESN has been proposed. Wout of ESN was optimized using the Grey Wolf algorithm and predictions of time series data were achieved using simplified training. The results indicated that the optimized time series prediction method exhibits superior prediction accuracy at a small sample size, compared with conventional prediction methods.

Design of polynomial echo state networks for time series prediction

Abstract Echo state networks (ESNs) have been widely used in the field of time series prediction. In conventional ESNs, the spectral radius of reservoir is always scaled to lower than 1 to satisfy the necessary condition for echo state property (ESP), while the sufficient condition is unregarded. Meanwhile, the output weights are always trained without considering the high order statistics of input signals. To solve above problems, the original ESN is extended to polynomial ESNs (PESNs) by employing the polynomial functions of input variables into output weights. Firstly, the reservoir of the PESN is built by the singular value decomposition (SVD) method. Secondly, the prime PESN (P-PESN) is implemented and its polynomial output weights are augmented by the high order statistics of inputs. Thirdly, the simplified PESN (S-PESN) is constructed by decomposing the polynomial output weights of the P-PESN into randomly generated polynomial nodes and tuned output weights. Furthermore, the regression matrix properties of P-PESN and S-PESN are theoretically analyzed, respectively. Finally, the simulation results show that the proposed PESNs obtain better performance than other methods in terms of prediction accuracy and learning speed.

The combination of circle topology and leaky integrator neurons remarkably improves the performance of echo state network on time series prediction.

Recently, echo state network (ESN) has attracted a great deal of attention due to its high accuracy and efficient learning performance. Compared with the traditional random structure and classical sigmoid units, simple circle topology and leaky integrator neurons have more advantages on reservoir computing of ESN. In this paper, we propose a new model of ESN with both circle reservoir structure and leaky integrator units. By comparing the prediction capability on Mackey-Glass chaotic time series of four ESN models: classical ESN, circle ESN, traditional leaky integrator ESN, circle leaky integrator ESN, we find that our circle leaky integrator ESN shows significantly better performance than other ESNs with roughly 2 orders of magnitude reduction of the predictive error. Moreover, this model has stronger ability to approximate nonlinear dynamics and resist noise than conventional ESN and ESN with only simple circle structure or leaky integrator neurons. Our results show that the combination of circle topology and leaky integrator neurons can remarkably increase dynamical diversity and meanwhile decrease the correlation of reservoir states, which contribute to the significant improvement of computational performance of Echo state network on time series prediction.