Chaotic Time Series Prediction Research Articles

Noise resistance of next-generation reservoir computing: a comparative study with high-order correlation computation

Reservoir computing (RC) methods have received more and more attention and applications in chaotic time series prediction with their simple structure and training method. Recently, the next-generation reservoir computing (NG-RC) method has been proposed by Gauthier et al. (Nat Commun 12:5564, 2021) with less training cost and better time series predictions. Nevertheless, in practice, available data on dynamic systems are contaminated with noise. Though NG-RC is shown highly efficient in learning and predicting, its noise resistance captivity is not clear yet, limiting its use in practical problems. In this paper, we study the noise resistance of the NG-RC method, taking the well-known denoising method, the high-order correlation computation (HOCC) method, as a reference. Both methods have similar procedures in respect of function bases and regression processes. With the simple ridge regression method, the NG-RC method has a strong noise resistance for white noise, even better than the HOCC method. Besides, the NG-RC method also shows a good prediction ability for small colored noise, while it does not provide correct reconstruct dynamics. In this paper, other than reconstruction parameters, four numerical indicators are used to check the noise resistance comprehensively, such as the training error, prediction error, prediction time, and auto-correlation prediction error, for both the short-time series and long climate predictions. Our results provide a systematic estimation of NG-RC’s noise resistance capacity, which is helpful for its applications in practical problems.

Multistep-Ahead Chaotic Time Series Prediction Based on Hierarchical Echo State Network With Augmented Random Features

Multistep-ahead chaotic time series prediction is a kind of highly nonlinear problem, which puts forward higher requirements both for the dynamical memory and nonlinearity of the model. Echo state network (ESN) is frequently employed in the realm of chaotic time series modeling and prediction, but the basic ESN has been proved to have an antagonistic trade-off between nonlinear transformation and memory capacity. To overcome this trade-off, a new architecture named hierarchical echo state network with augmented random features (HESN-ARF) is proposed. On the basis of traditional linear random projection, the proposed HESN-ARF further leverages nonlinear kernel transformation to construct augmented random features, which can enable the linear and nonlinear properties to be fully represented. Moreover, the HESN-ARF utilizes low-rank kernel approximation to further reduce the computational cost, preserving the advantage of efficient modeling as much as possible while ensuring the capacities of nonlinear transformation and dynamical memory simultaneously. The proposed HESN-ARF can mine and learn the latent evolution patterns hidden in the dynamic system layer by layer through the hierarchical strategy, and achieves excellent performance in multi-step-ahead chaotic time series prediction, as demonstrated by experimental findings on two synthetic chaotic systems and a real-world meteorological dataset.

Chaotic Time Series Prediction of Multi‐Dimensional Nonlinear System Based on Bidirectional LSTM Model

AbstractThe current work proposes a hybrid data‐driven model—Convolutional bidirectional long–short term memory (CNN‐BLSTM) for predicting chaotic behavior of three‐coupled Duffing oscillator nonlinear system, in which the CNN is for efficiently extracting the more robust and informative representations of chaotic sequences while the BLSTM is for holding the long‐term dependencies combining the past and future contexts. Different from traditional analytical and numerical approaches, the proposed prediction model features the benefit of focusing on the measured data solely without extensive professional domain knowledge. Additionally, three more recurrent neural network (RNN) models, including simple RNNs, stack LSTMs, and BLSTM, are built and comparisons of generalization performances to the CNN‐BLSTM are conducted. From the findings so far, the CNN‐BLSTM is able to learn the pattern of chaotic time sequence data with less training time and apply the acquired knowledge to the unseen dataset with lower errors. Moreover, the current work decently demonstrates that the proposed model outperforms other three models in terms of stability at different noise levels from two evaluation criteria. The CNN‐BLSTM provides useful guidance for the consideration of predicting multi‐dimensional nonlinear chaotic behavior.

Online quantum time series processing with random oscillator networks

Reservoir computing is a powerful machine learning paradigm for online time series processing. It has reached state-of-the-art performance in tasks such as chaotic time series prediction and continuous speech recognition thanks to its unique combination of high computational power and low training cost which sets it aside from alternatives such as traditionally trained recurrent neural networks, and furthermore is amenable to implementations in dedicated hardware, potentially leading to extremely compact and efficient reservoir computers. Recently the use of random quantum systems has been proposed, leveraging the complexity of quantum dynamics for classical time series processing. Extracting the output from a quantum system without disturbing its state too much is problematic however, and can be expected to become a bottleneck in such approaches. Here we propose a reservoir computing inspired approach to online processing of time series consisting of quantum information, sidestepping the measurement problem. We illustrate its power by generalizing two paradigmatic benchmark tasks from classical reservoir computing to quantum information and introducing a task without a classical analogue where a random system is trained to both create and distribute entanglement between systems that never directly interact. Finally, we discuss partial generalizations where only the input or only the output time series is quantum.

Enhanced FPGA implementation of Echo State Networks for chaotic time series prediction

The Echo State Network (ESN) is one of the most used machine learning methods for predicting chaotic time series. The topology of an ESN comprises three layers, namely: input, hidden, and output layer. In this work, different ESN topologies are analyzed by modifying the structure and number of internal connections in the hidden layer, which neurons are connected randomly. The ESN topologies are analyzed to have 2%, 6%, 20%, 40%,…, and up to 100% of internal connections. The prediction error is associated to the Mean Squared Error (MSE), which results have similar values for all ESN topologies, as long as the Echo state property is guaranteed. It is highlighted that the prediction horizon obtained with 2% of internal connections, is more significant than that obtained with 100%, providing a prediction of 80 and 60 steps-ahead, respectively. The prediction results significantly impact the implementation in a field-programmable gate array (FPGA), so that an enhanced hardware realization is introduced herein by reducing the use of multipliers by more than 90% compared with a fully connected topology, without significantly affecting the measure of the prediction horizon and MSE. Another contribution is the FPGA implementation of the activation function that is an hyperbolic tangent function for which different alternatives are shown to minimize the error of the approximated function without significantly increasing the implementation cost or latency.

Enhanced Performance of Reservoir Computing Using Multiple Self-Injection and Mutual Injection VCSELs

We propose and numerically demonstrate two reservoir computing (RC) schemes using multiple vertical-cavity surface-emitting lasers (VCSELs) with different external injections, i.e., self-injection and mutual injection. The virtual nodes are obtained from the orthogonal polarization modes of multiple VCSELs. We evaluate the performance of our proposed RC systems by using the chaotic time-series prediction and memory capacity (MC). Similar to edge-emitting lasers (EELs), the multiple VCSEL-based RC scheme exhibits better computing performance compared with a single VCSEL using several delay times. More interestingly, numerical results show that our proposed scheme using dual-mode is superior to that using single-mode under the scenario of the same number of virtual nodes, and the scheme using mutual injection VCSELs outperforms that using self-injection VCSELs, in which the coupling mechanism between both polarization modes may play an important role. Further, the effects of different polarized feedback/coupling configurations on both reservoir schemes are discussed in terms of the prediction performance and MC. Additionally, we demonstrate for the first time that the polarization mode competition significantly affects the parallel VCSEL-based RC performance through analyzing its dependence on the feedback phase.

An Efficient Filtering Algorithm against Impulse Noise in Communication Systems

<p>The kernel adaptive filter (KAF), which processes data in the reproducing kernel Hilbert space (RKHS), can improve the performance of conventional adaptive filters in nonlinear systems. However, the presence of impulse noise can seriously degrade the performance of KAF. In this paper, we propose a kernel modified-sign least-mean-square algorithm (KMSLMS) to mitigate the impact of impulse noise in communication systems. Moreover, we apply the nearest-instance-centroid estimation (NICE) algorithm to reduce the computational complexity of our KMSLMS algorithm, called the NICE-KMSLMS algorithm. Finally, computer simulations were used to evaluate the effectiveness of our proposed method. Compared with the conventional kernel least-mean-square algorithm (KLMS), our proposed method can improve the testing mean-squared error (MSE) by 2.32 dB and 7.39 dB for the nonlinear channel equalization and Mackey-Glass chaotic time series prediction problems, respectively. Furthermore, the testing MSE degradation caused by combining the NICE algorithm with our KMSLMS algorithm is negligible but can save about 55% computational cost in terms of the required mean size.</p> <p> </p>

Forecasting the chaotic dynamics of external cavity semiconductor lasers.

Chaotic time series prediction has been paid intense attention in recent years due to its important applications. Herein, we present a single-node photonic reservoir computing approach to forecasting the chaotic behavior of external cavity semiconductor lasers using only observed data. In the reservoir, we employ a semiconductor laser with delay as the sole nonlinear physical node. By investigating the effect of the reservoir meta-parameters on the prediction performance, we numerically demonstrate that there exists an optimal meta-parameter space for forecasting optical-feedback-induced chaos. Simulation results demonstrate that using our method, the upcoming chaotic time series can be continuously predicted for a time period in excess of 2 ns with a normalized mean squared error lower than 0.1. This proposed method only utilizes simple nonlinear semiconductor lasers and thus offers a hardware-friendly approach for complex chaos prediction. In addition, this work may provide a roadmap for the meta-parameter selection of a delay-based photonic reservoir to obtain optimal prediction performance.

Improving Deep Echo State Network with Neuronal Similarity-Based Iterative Pruning Merging Algorithm

Recently, a layer-stacked ESN model named deep echo state network (DeepESN) has been established. As an interactional model of a recurrent neural network and deep neural network, investigations of DeepESN are of significant importance in both areas. Optimizing the structure of neural networks remains a common task in artificial neural networks, and the question of how many neurons should be used in each layer of DeepESN must be stressed. In this paper, our aim is to solve the problem of choosing the optimized size of DeepESN. Inspired by the sensitive iterative pruning algorithm, a neuronal similarity-based iterative pruning merging algorithm (NS-IPMA) is proposed to iteratively prune or merge the most similar neurons in DeepESN. Two chaotic time series prediction tasks are applied to demonstrate the effectiveness of NS-IPMA. The results show that the DeepESN pruned by NS-IPMA outperforms the unpruned DeepESN with the same network size, and NS-IPMA is a feasible and superior approach to improving the generalization performance of DeepESN. The newly proposed method has broad application prospects in real-time systems.

An optimized wavelet neural networks using cuckoo search algorithm for function approximation and chaotic time series prediction

Although the practicability of using wavelet neural networks (WNNs) in nonlinear function approximation has been addressed extensively, selecting the optimal number of hidden nodes and their appropriate initial locations remains a great challenge for WNNs’ initialization. The cuckoo search algorithm (CSA) is used in this study for optimizing WNNs. The position of the cuckoo eggs represents the translation of the wavelet hidden nodes, which are optimized based on the egg-laying and breeding strategy of cuckoos. The solutions from the CSA are assigned as initial translation vectors for the WNNs and subsequently evaluated on a few benchmarking functions and real-world applications. Performance assessment demonstrates its superior approximation capability than the existing methods used for WNNs initialization.

Interval type-2 fuzzy neural networks with asymmetric MFs based on the twice optimization algorithm for nonlinear system identification

This paper proposes a novel algorithm twice optimization for interval type-2 fuzzy neural networks with asymmetric membership functions (TOIT2FNN-AMF), for nonlinear system identification problems. The proposed TOIT2FNN-AMF uses an asymmetric Gaussian interval type-2 membership function to enhance the network’s ability to describe and solve nonlinear and uncertain problems. The twice optimization algorithm consists of structure learning and parameter learning. Firstly, this paper proposes a multi-strategy adaptive differential evolution (MSADE) algorithm as the first optimization algorithm, which is used to determine the structure and the initial values of the parameters of the TOIT2FNN-AMF. It applies the root mean square error (RMSE) of the TOIT2FNN-AMF as the fitness function to determine the structure (number of rules) and initial parameters of the IT2FNN by searching for the RMSE values under different structures. When the fitness value reaches the minimum, that is, the RMSE value of the TOIT2FNN-AMF, the corresponding number will become the optimal one of fuzzy rules of the TOIT2FNN-AMF. Then, the second optimization algorithm of the TOIT2FNN-AMF turns into a hybrid optimization algorithm composed of an adaptive moment estimation (Adam) algorithm and recursive least squares (RLS) algorithm. Adam is used to optimize the antecedent parameters of TOIT2FNN-AMF rules, so as to maintain rapid convergence without generating oscillation during the training process; RLS is used to optimize the consequent parameters of TOIT2FNN-AMF rules, so that the network parameters can be optimized rapidly. In this way, the problems of excessive parameters to be adjusted and excessive slow convergence of the network can be solved. Finally, this paper evaluates the proposed TOIT2FNN-AMF by testing on problems of nonlinear system identification and chaotic time-series prediction. The simulation results are compared with those of similar methods in the existing literatures, which demonstrates that the proposed TOIT2FNN-AMF model yields a lower RMSE value and a simpler network structure than the other type-2 fuzzy neural networks (T2FNNs).

Driver-Centric Velocity Prediction With Multidimensional Fuzzy Granulation

Dear Editor, This letter deals with a real-world problem regarding chaotic time series prediction, where a driver-centric velocity prediction model is presented for vehicle intelligent control and advanced driver assistance, i.e., multi-dimension fuzzy predictor. Inspired by fuzzy granulation technology, a finite-state Markov chain (MC) is reinforced to capture probabilities of the transitions between velocity and acceleration and present signals that vary in a continuous range. The predictability of the multi-dimensional fuzzy predictor is examined by comparing two existing MC-based predictors over the two laboratory cycles and one virtual driving cycle, both of which have high accuracy.

Improving the processing rate of VCSEL-based reservoir computing through adopting variable polarization information injection

A novel information injection method is proposed for improving the information processing rate in reservoir computing (RC) systems based on vertical-cavity surface-emitting lasers (VCSELs), and its effectiveness is numerically verified via Santa-Fe chaotic time series prediction task and waveform recognition task. For VCSEL-based RC systems, conventional information injection (CII) method is usually adopted, namely the polarization direction of the injection light loaded by the information is along either X-polarization component (X-PC) or Y-polarization component (Y-PC) of VCSEL. In this work, the polarization direction of the injection light loaded by the information is set at 45-degree deviated from the X-PC of the VCSEL, which is named as variable polarization information injection (VPII) method. By utilizing the VCSEL-based RC to deal with two different scenarios of single task and two tasks in parallel, the numerical results demonstrate that after adopting VPII to replace CII, the processing rate can be doubled while the system performance maintains the same level.

On the learning machine in quaternionic domain and its application

There are various high-dimensional engineering and scientific applications in communication, control, robotics, computer vision, biometrics, etc.; where researchers are facing problem to design an intelligent and robust neural system which can process higher dimensional information efficiently. In various literatures, the conventional real-valued neural networks are tried to solve the problem associated with high-dimensional parameters, but the required network structure possesses high complexity and are very time consuming and weak to noise. These networks are also not able to learn magnitude and phase values simultaneously in space. The quaternion is the number, which possesses the magnitude in all four directions and phase information is embedded within it. This paper presents a learning machine with a quaternionic domain neural network that can finely process magnitude and phase information of high dimension data without any hassle. The learning and generalisation capability of the proposed learning machine is presented through 3D linear transformations, 3D face recognition and chaotic time series predictions (Lorenz system and Chua's circuit) as benchmark problems, which demonstrate the significance of the work.

Temporal Convolutional Networks with RNN approach for chaotic time series prediction

The prediction of chaotic time series, which constitutes many systems in the field of science and engineering, has recently become the focus of attention of researchers. Chaotic time series prediction is making future predictions about these systems using previously observed data for a nonlinear chaotic system with a known initial condition. Chaotic time series prediction can be applied in many fields such as weather forecasting, finance and stock markets. Many disciplines work on solving time series prediction problem, ranging from forecasting weather events days in advance to traders predicting the future of stocks. In recent studies, it has been observed that hybrid deep neural network methods give better performance in solving time series prediction problems and have gained popularity in order to benefit from the advantages of more than one method in solving such problems. In this study, a hybrid deep neural network architecture is proposed for chaotic time series prediction. The used hybrid approach includes both temporal convolutional network to extract low level features from input and recurrent neural network layers such as long short-term memory and gated recurrent units to capture temporal information. Simulations were carried out on nine different chaotic time series dataset which are obtained from Lorenz, Rössler and a Lorenz-like chaotic equation sets, and twenty-one electrocardiogram (ECG) recordings of patients with arrhythmias. In the benchmark study, in which twelve different methods, including classical machine learning, deep neural network and hybrid models were used, the proposed model achieved the best prediction performance with an average root-mean-square error (RMSE) value of 0.0022 for chaotic dataset and 0.0082 for ECG arrhythmia dataset. Performance evaluation metrics show that the proposed hybrid architecture can compete with the models in state-of-the-art studies in chaotic time series prediction.

Parameterizing echo state networks for multi-step time series prediction

Prediction of multi-dimensional time-series data, which may represent such diverse phenomena as climate changes or financial markets, remains a challenging task in view of inherent nonlinearities and non-periodic behavior. In contrast to other recurrent neural networks, echo state networks (ESNs) are attractive for (online) learning due to lower requirements w.r.t.training data and computational power. However, the randomly-generated reservoir renders the choice of suitable hyper-parameters as an open research topic. We systematically derive and exemplarily demonstrate design guidelines for the hyper-parameter optimization of ESNs. For the evaluation, we focus on the prediction of chaotic time series, an especially challenging problem in machine learning. Our findings demonstrate the power of a hyper-parameter-tuned ESN when auto-regressively predicting time series over several hundred steps. We found that ESNs’ performance improved by 85.1%-99.8% over an already wisely chosen default parameter initialization. In addition, the fluctuation range is considerably reduced such that significantly worse performance becomes very unlikely across random reservoir seeds. Moreover, we report individual findings per hyper-parameter partly contradicting common knowledge to further, help researchers when training new models.

A novel non-linear neuron model based on multiplicative aggregation in quaternionic domain

The learning algorithm for a three-layered neural structure with novel non-linear quaternionic-valued multiplicative (QVM) neurons is proposed in this paper. The computing capability of non-linear aggregation in the cell body of biological neurons inspired the development of a non-linear neuron model. However, unlike linear neuron models, most non-linear neuron models are built on higher order aggregation, which is more mathematically complex and difficult to train. As a result, building non-linear neuron models with a simple structure is a difficult and time-consuming endeavor in the neurocomputing field. The concept of a QVM neuron model was influenced by the non-linear neuron model, which has a simple structure and the great computational ability. The suggested neuron’s linearity is determined by the weight and bias associated with each quaternionic-valued input. Non-commutative multiplication of all linearly connected quaternionic input-weight terms accommodates the non-linearity. To train three-layered networks with QVM neurons, the standard quaternionic-gradient-based backpropagation (QBP) algorithm is utilized. The computational and generalization capabilities of the QVM neuron are assessed through training and testing in the quaternionic domain utilizing benchmark problems, such as 3D and 4D chaotic time-series predictions, 3D geometrical transformations, and 3D face recognition. The training and testing outcomes are compared to conventional and root-power mean (RPM) neurons in quaternionic domain using training–testing MSEs, network topology (parameters), variance, and AIC as statistical measures. According to these findings, networks with QVM neurons have greater computational and generalization capabilities than networks with conventional and RPM neurons in quaternionic domain.

Misestimate of the performance in VCSEL-based reservoir computing systems with optical information injection by high surface reflectivity.

In reservoir computing (RC) systems based on semiconductor lasers (SLs), the information that must be processed usually enters the reservoir through optical injection. Part of the injection information directly reflected by the front facet of the SLs is inevitably hybridized into the output of the SLs and contributes to the state of virtual nodes. For an RC system based on vertical-cavity surface-emitting lasers (VCSELs), the proportion of the reflected information coupled to the laser output is relatively huge due to the high surface reflectivity. Thus the influence of the directly reflected information will be much more obvious. Using a Santa Fe chaotic time series prediction task and waveform recognition task, we theoretically investigate the influence of high front facet reflectivity on the evaluation of the performance of a VCSEL-based RC system with optical information injection. The simulation results demonstrate that, after taking the directly reflected information into account, a lower error rate is obtained for each benchmark task. The physical mechanism to misestimate the RC performance has been studied through memory correlation and a statistical histogram of virtual node states.

Multi-Attn BLS: Multi-head attention mechanism with broad learning system for chaotic time series prediction

The observational 1-D signals available for realizing the highly accurate intrinsic attractor fitting of deep learning network approaches are often insufficient because of the complexity and nonlinearity of chaotic time series. Unlike deep models, a broad learning system (BLS) with the attention mechanism exhibits a unique and preeminent pattern prediction ability. Thus, this system has been applied as a practical trend in many fields. However, the application of multi-head attention fused manifold broad learning architecture to chaotic time series prediction remains inadequate. Thus, a multi-head attentional BLS (Multi-Attn BLS) for chaotic time series prediction is proposed in this study to improve the prediction accuracy of chaotic time series further. Our model develops a novel framework that combines the high computational efficiency of broad learning with the multi-head attention mechanism. First, the received data are reconstructed into fixed-size tuples. The multidimensional arrays with embedding dimensions and time delay are used as the input to a broad learning network. Subsequently, a robust BLS with a spatiotemporal multi-head attention mechanism is developed to depict the internal dynamic evolution. The Multi-Attn BLS model can capture key spatiotemporal feature information and achieve high predictive performance. It also has a good generalization ability in practical nonlinear complex systems. Comparative experiments with the traditional long short-term memory (LSTM) network and the primitive BLS show that its computing speed and generalization ability are improved. Furthermore, the network is good at capturing the spatiotemporal features of the sequence because of the multi-head attention mechanism. The experimental results show that our model outperforms BLS, ridge regression, and LSTM on the four main evaluation indicators (root mean square error, root mean square percentage error, mean absolute error, and mean absolute percentage error) in predicting classical systems (Lorenz and Rossler systems). Moreover, the model has an excellent prediction effect in the real-world chaotic system of sea clutter.

Online Tracking of Maneuvering Target Trajectory Based on Chaotic Time Series Prediction.

Online prediction of maneuvering target trajectory is one of the most popular research directions at present. Specifically, the primary factors balancing, between prediction accuracy and response time, will give the research substance. This paper presents an online trajectory prediction algorithm based on small sample chaotic time series (OTP-SSCT). First, we optimize in terms of data breadth. The dynamic split window is built according to the motion characteristics of the maneuvering target, thus realizing trajectory segmentation and constructing a small sample chaotic time series prediction set. Second, since fully considering the motion patterns of maneuvering targets, we introduce the spatiotemporal features into the particle swarm optimization (PSO) model identification algorithm, which improves the identification sensitivity of key trajectory data points. Furthermore, we propose a feedback optimization strategy of residual compensation to correct the trajectory prediction values to improve the prediction accuracy. For the initial value sensitivity problem of the PSO model identification algorithm, we propose a new initial population strategy, which improves the effectiveness of initial parameters on model identification. Through simulation experiment analysis, it is verified that the proposed OTP-SSCT algorithm achieves better prediction accuracy and faster response time.