Chaotic Time Series Prediction Research Articles

Prediction of chaotic time series based on the fractional-order maximum correntropy criterion algorithm

Recently, adaptive filters have been widely used to perform the prediction of chaotic time series. Generally, the Gaussian noise is considered for the system noise. However, many non-Gaussian noises, e.g., impulse noise and alpha noise, exist in real systems. Adaptive filters are therefore required to reduce such non-Gaussian noises for practical applications. For improving the robustness against non-Gaussian noise, the maximum correntropy criterion (MCC) is successfully used to derive various robust adaptive filters. In these robust adaptive filters, the steepest ascent method based on the first-order derivative is generally utilized to construct the weight update form. It is well known that the traditional derivative can be generalized by the fractional-order derivative effectively. Therefore, to further improve the performance of adaptive filters based on the MCC, the fractional-order derivative is applied to the MCC-based algorithm, generating a novel fractional-order maximum correntropy criterion (FMCC) algorithm. Under the non-Gaussian noises, the proposed FMCC algorithm can be applied to predicting the chaotic time series effectively. In the proposed FMCC algorithm, the weight update form is constructed by using a combination of the first-order derivative based term and the fractional-order derivative based term. The Riemann-Liouville definition is utilized for calculating the fractional-order derivative in the proposed FMCC algorithm. The order of the fractional-order derivative is a crucial parameter of the proposed FMCC algorithm. However, it is difficult to obtain the optimal fractional order for different nonlinear systems theoretically. Therefore, the influence of the fractional order on the prediction performance is determined by trials for different nonlinear systems. The appropriate fractional order corresponds to the optimum of prediction accuracy, and can be chosen in advance. Simulations in the context of prediction of Mackey-Glass time series and Lorenz time series demonstrate that in the case of non-Gaussian noises the proposed FMCC algorithm achieves better prediction accuracy and faster convergence rate than the least mean square (LMS) algorithm, the MCC algorithm, and the fractional-order least mean square (FLMS) algorithm. In addition, the computational complexity of different filters is compared with each other under the example of the prediction of Marckey-Glass time series by using mean consumed time. It can be found that the computational complexity of FMCC algorithm is higher than those of the MCC and the LMS algorithms, but only slightly higher than that of the FLMS algorithm. As a result, comparing with other filters, the FMCC algorithm can improve the prediction performances of chaotic time series at the cost of the increasing computational complexity.

Adaptive Levenberg-Marquardt Algorithm Based Echo State Network for Chaotic Time Series Prediction

Echo state networks (ESNs) have wide applications in chaotic time series prediction. In the ESN, if the smallest singular value of the reservoir state matrix is infinitesimal, the ill-posed problem might occur during the training process. To overcome this problem, an adaptive Levenberg–Marquardt (LM) algorithm-based echo state network (ALM-ESN) is developed. In the developed ALM-ESN, a new adaptive damping term is introduced into the LM algorithm. The adaptive factor is amended by the trust region technique, furthermore, convergence analysis, and stability analysis are performed. Moreover, to make the inputs fall within the active region of the activation function and improve the learning speed, a weight initialization method using linear algebra is deployed to determine the appropriate input weights and reservoir weights. Simulations demonstrate that the ALM-ESN can overcome the ill-posed problem. Furthermore, it exhibits better performance and robustness for chaotic time series prediction than some other existing methods.

Chaotic time series prediction based on robust extreme learning machine

Chaos is seemingly irregular and analogous to random movement happening in a determinative system in nature,and more and more types and numbers of time series with chaotic characteristics are obtained from the actual systems,such as atmospheric circulation,temperature,rainfall,sunspots,and the Yellow River flow.The chaotic time series prediction has become a research hotspot in recent years.Because neural network can be strongly approximated nonlinearly,it has better prediction performance in the chaotic time series modeling.Extreme learning machine is a kind of neural network, and it is widely used due to its simple structure,high learning efficiency and having global optimal solution.Extreme learning machine initializes the input weight randomly and just adjusts the output weight in the training process,in order to be able to obtain the global optimal solution,so it has faster convergence speed and can overcome the disadvantage of gradient vanishing.Due to the above advantages,in recent years,the improved algorithms of the extreme learning machine have been developed rapidly.However,the traditional training methods of extreme learning machine have very poor robustness and can be affected easily by noise and outliers.And in practical applications,the time series are often contaminated by noise and outliers,so it is important to improve the forecasting model robustness and reduce the influence of noise and abnormal points to obtain better prediction accuracy.In this paper,a robust extreme learning machine is proposed in a Bayesian framework to solve the problem that outliers exist in the training data set.Firstly,the input samples are mapped onto the high-dimensional space,and the output weight of the extreme learning machine is used as the parameter to be estimated,then the proposed model utilizes the more robust Gaussian mixture distribution as the likelihood function of the model output.The marginal likelihood of the model output is analytically intractable for the Gaussian mixture distribution,so a variational procedure is introduced to realize the parameter estimation.In the cases of different noise levels and the different numbers of outliers,the proposed model is compared with the other prediction models.The experimental results of Lorenz,Rossler and Sunspot-Runoff in the Yellow River time series with outliers and noise demonstrate that the proposed robust extreme learning machine model could obtain a better prediction accuracy.The proposed robust extreme learning machine not only has the strong capability of the nonlinear approximation but also can learn the model parameters automatically and has strong robustness.At the same time,the time complexities of different models are compared and the convergence of the proposed model is analyzed at the end of the paper.

Kernel Adaptive Filters With Feedback Based on Maximum Correntropy

This paper presents novel kernel adaptive filters with feedback, namely, kernel recursive maximum correntropy with multiple feedback (KRMC-MF) and its simplified version, a linear recurrent kernel online learning algorithm based on maximum correntropy criterion (LRKOL-MCC). In LRKOL-MCC and KRMC-MF, single output and multiple outputs based on single delay are utilized to construct their feedback structure, respectively. Compared with the minimum mean square error criterion, the maximum correntropy criterion (MCC) adopted by LRKOL-MCC and KRMC-MF captures higher order statistics of errors. The proposed filters are, therefore, robust against outliers. Therefore, the past information can be reused to improve filtering performance in terms of the steady-state mean square error. The convergence characteristics of the filter parameters in LRKOL-MCC and KRMC-MF are also derived. Simulations on chaotic time-series prediction and nonlinear regression illustrate the desirable accuracy and robustness of the proposed filters.

Chaotic time series prediction based on brain emotional learning model and self-adaptive genetic algorithm

Chaos phenomenon is one of the most important physical phenomena, which has significant effects on one's production and life. Therefore, it is indispensable to find out the regularity of chaotic time series from a chaotic system for weather forecasting, space missions, alarm systems, etc. Although various models and learning algorithms have been developed to predict chaotic time series, many traditional methods suffer drawbacks of high computational complexity, slow convergence speed, and low prediction accuracy, due to extremely complex dynamic characteristics of chaotic systems. In this paper, a brain-inspired prediction model, i.e., brain emotional learning (BEL) model combined with self-adaptive genetic algorithm (AGA) is proposed. The establishment of BEL model is inspired by the neurobiology research, which has been put forward by mimicking the high-speed emotional learning mechanism between amygdala and orbitofrontal cortex in mammalian brain, it has advantages of lowcomputational complexity and fast learning. The BEL model employs reward-based reinforcement learning to adjust the weights of amygdala and orbitofrontal cortex. However, the reward-based method is modelsensitive and hard to generalize to other issues. To improve the performance of BEL model, AGA-BEL is proposed for chaotic prediction, in which the AGA is employed for parameter optimization. Firstly, weights and biases of orbitofrontal cortex and amygdala in BEL model are distributed to chromosomal gene sequence for optimization. Secondly, fitness function is employed to adjust the weights of amygdale and orbitofrontal cortex by self-adaptive crossover and mutation operations Therefore, the parameter optimization problem is transformed into a function optimization problem in the search space. Finally, the best chromosome that represents the best combination of weights and biases for BEL model is chosen, which is used for chaotic prediction. Prediction experiments on the benchmark Lorenz chaotic time series and a real-world chaotic time series of geomagnetic activity Dst index are performed. The experimental results and numerical analysis show that the proposed AGA-BEL prediction model achieves lower mean absolute deviation, mean square error, mean absolute percentage error, and higher correlation coefficient than the original BEL, levenberg marquardt-back propagation (LM-BP) and multilayer perceptron-back propagation (MLP-BP). Meanwhile, the BEL-based models take less computational time than the traditional BP-based models. Therefore, the proposed AGA-BEL model possesses the advantages of fast learning and low computational complexity of BEL model as well as the global optimum solution of AGA. It is superior to other traditional methods in terms of prediction precision, execution speed, and stability, and it is suited for online prediction in fast-varying environments.

Transfer Learning in Adaptive Filters: The Nearest Instance Centroid-Estimation Kernel Least-Mean-Square Algorithm

We propose a novel nearest-neighbors approach to organize and curb the growth of radial basis function network in kernel adaptive filtering (KAF). The nearest-instance-centroid-estimation (NICE) kernel least-mean-square (KLMS) algorithm provides an appropriate time-space tradeoff with good performance. Its centers in the input/feature space are organized by quasi-orthogonal regions for greatly simplified filter evaluation. Instead of using all centers to evaluate/update the function approximation at every new point, a linear search among the iteratively-updated centroids determines the partial function to be used, naturally forming locally-supported partial functionals. Under this framework, partial functionals that compose the adaptive filter are quickly stored/retrieved based on input, each corresponding to a specialized “spatial-band” subfilter. The filter evaluation becomes the update of one of the subfilters, creating a content addressable filter bank (CAFB). This CAFB is incrementally updated for new signal applications with mild constraints, always using the past-learned partial filter sums, opening the door for transfer learning and significant efficiency for new data scenarios, avoiding training from scratch as have been done since the invention of adaptive filtering. Using energy conservation relation, we show the sufficient condition for mean square convergence of the NICE-KLMS algorithm and establish the upper and lower bounds of steady-state excess-mean-square-error (EMSE). Simulations on chaotic time-series prediction demonstrate similar levels of accuracy as existing methods, but with much faster computation involving fewer input samples. Simulations on transfer learning using both synthetic and real-world data demonstrate that NICE CAFB can leverage previously learned knowledge to related task or domain.

The Automatic Model Selection and Variable Width RBF Neural Networks for Chaotic Time Series Prediction

The Automatic Model Selection and Variable Width RBF Neural Networks for Chaotic Time Series Prediction

Research on Traffic Flow Prediction in the Big Data Environment Based on the Improved RBF Neural Network

This paper proposes an optimized prediction algorithm of radial basis function neural network based on an improved artificial bee colony (ABC) algorithm in the big data environment. The algorithm first uses crossover and mutation operators of the differential evolution algorithm to replace the search strategy of employed bees in the ABC algorithm, then improves the search strategy of onlookers in the ABC algorithm to produce an optimal candidate food source near the population. The algorithm can better balance local and global searching capability. To verify the efficiency of this algorithm in the big data environment, apply it to Lozi and Tent chaotic time series and measured traffic flow time series, and then compare it with K-nearest neighbor model, radial basis function (RBF) neural network model, improved back propagation neural network model, and RBF neural network based on a cloud genetic algorithm model. The experimental results indicate that the accuracy of prediction for Lozi and Tent chaotic time series and the measured traffic flow improves greatly in the big data environment using the proposed algorithm, which proves the effectiveness of the proposed algorithm in predicting traffic flow time series.

Chaotic time series prediction for glucose dynamics in type 1 diabetes mellitus using regime-switching models

In patients with type 1 diabetes mellitus (T1DM), glucose dynamics are influenced by insulin reactions, diet, lifestyle, etc., and characterized by instability and nonlinearity. With the objective of a dependable decision support system for T1DM self-management, we aim to model glucose dynamics using their nonlinear chaotic properties. A group of patients was monitored via continuous glucose monitoring (CGM) sensors for several days under free-living conditions. We assessed the glycemic variability (GV) and chaotic properties of each time series. Time series were subsequently transformed into the phase-space and individual autoregressive (AR) models were applied to predict glucose values over 30-minute and 60-minute prediction horizons (PH). The logistic smooth transition AR (LSTAR) model provided the best prediction accuracy for patients with high GV. For a PH of 30 minutes, the average values of root mean squared error (RMSE) and mean absolute error (MAE) for the LSTAR model in the case of patients in the hypoglycemia range were 5.83 ( ± 1.95) mg/dL and 5.18 ( ± 1.64) mg/dL, respectively. For a PH of 60 minutes, the average values of RMSE and MAE were 7.43 ( ± 1.87) mg/dL and 6.54 ( ± 1.6) mg/dL, respectively. Without the burden of measuring exogenous information, nonlinear regime-switching AR models provided fast and accurate results for glucose prediction.

A Quantized Kernel Learning Algorithm Using a Minimum Kernel Risk-Sensitive Loss Criterion and Bilateral Gradient Technique

Recently, inspired by correntropy, kernel risk-sensitive loss (KRSL) has emerged as a novel nonlinear similarity measure defined in kernel space, which achieves a better computing performance. After applying the KRSL to adaptive filtering, the corresponding minimum kernel risk-sensitive loss (MKRSL) algorithm has been developed accordingly. However, MKRSL as a traditional kernel adaptive filter (KAF) method, generates a growing radial basis functional (RBF) network. In response to that limitation, through the use of online vector quantization (VQ) technique, this article proposes a novel KAF algorithm, named quantized MKRSL (QMKRSL) to curb the growth of the RBF network structure. Compared with other quantized methods, e.g., quantized kernel least mean square (QKLMS) and quantized kernel maximum correntropy (QKMC), the efficient performance surface makes QMKRSL converge faster and filter more accurately, while maintaining the robustness to outliers. Moreover, considering that QMKRSL using traditional gradient descent method may fail to make full use of the hidden information between the input and output spaces, we also propose an intensified QMKRSL using a bilateral gradient technique named QMKRSL_BG, in an effort to further improve filtering accuracy. Short-term chaotic time-series prediction experiments are conducted to demonstrate the satisfactory performance of our algorithms.

A new chaotic time series hybrid prediction method of wind power based on EEMD-SE and full-parameters continued fraction

The wind power time series always exhibits nonlinear and non-stationary features, which make it very difficult to predict accurately. In this paper, a new chaotic time series prediction model of wind power based on ensemble empirical mode decomposition-sample entropy (EEMD-SE) and full-parameters continued fraction is proposed. In this proposed method, EEMD-SE technique is used to decompose original wind power series into a number of subsequences with obvious complexity differences. The forecasting model of each subsequence is created by full-parameters continued fraction. On the basis of the inverse difference quotient continued fraction, the full-parameters continued fraction model is proposed. The parameters of model are optimized by the primal dual state transition algorithm (PDSTA). The effectiveness of the proposed approach is demonstrated with practical hourly data of wind power generation in Xinjiang. A comprehensive error analysis is carried out to compare the performance with other approaches. The forecasting results show that forecast improvement is observed based on EEMD-SE and full-parameters continued fraction model.

Performance improvement via bagging in probabilistic prediction of chaotic time series using similarity of attractors and LOOCV predictable horizon

Recently, we have presented a method of probabilistic prediction of chaotic time series. The method employs learning machines involving strong learners capable of making predictions with desirably long predictable horizons, where, however, usual ensemble mean for making representative prediction is not effective when there are predictions with shorter predictable horizons. Thus, the method selects a representative prediction from the predictions generated by a number of learning machines involving strong learners as follows: first, it obtains plausible predictions holding large similarity of attractors with the training time series and then selects the representative prediction with the largest predictable horizon estimated via LOOCV (leave-one-out cross-validation). The method is also capable of providing average and/or safe estimation of predictable horizon of the representative prediction. We have used CAN2s (competitive associative nets) for learning piecewise linear approximation of nonlinear function as strong learners in our previous study, and this paper employs bagging (bootstrap aggregating) to improve the performance, which enables us to analyze the validity and the effectiveness of the method.

Implementation of regularization for separable nonlinear least squares problems

This paper considers a class of separable nonlinear least squares problems in which a model can be represented as a linear combination of nonlinear functions. A regularized nonlinear parameter optimization approach is presented for coping with the potential ill-conditioned problem of parameter divergence. Together with a regularization parameter detection technique, Tikhonov regularization and truncated singular value decomposition are utilized in the estimation of the linear parameters if the nonlinear parameters are changed during the parameter optimization process, which centers on a nonlinear parameter search using the Levenberg-Marquardt algorithm. Benefiting from the regularization in parameter optimization, the potential ill-conditioned issue can be avoided, and the multi-step-ahead forecasting accuracy of the estimated model may be largely improved. The usefulness of this approach is illustrated by means of a chaotic time-series prediction and nonlinear industrial process modeling.

Robust echo state networks based on correntropy induced loss function

In this paper, a robust echo state network with correntropy induced loss function (CLF) is presented. CLF is robust to outliers through the mechanism of correntropy which is widely applied in information theoretic learning. The proposed method can improve the anti-noise capacity of echo state network and overcome its problem of being sensitive outliers which are prevalent in real-world tasks. The echo state network with CLF inherits the basic architecture of echo state network, but replaces the commonly used mean square error (MSE) criterion with CLF. The stochastic gradient descent method is adopted to optimize the objective function. The proposed method is subsequently verified in nonlinear system identification and chaotic time-series prediction. Experimental results demonstrate that our method is robust to outliers and outperforms the echo state networks with Bayesian regression and Huber loss function.

A learning and niching based backtracking search optimisation algorithm and its applications in global optimisation and ANN training

A backtracking search optimisation algorithm that uses historic population information for learning was proposed recently for solving optimisation problems. However, the learning ability and the robustness of this algorithm remain relatively poor. To improve the performance of the backtracking search algorithm (BSA), a modified backtracking search optimisation algorithm (MBSA), based on learning and niching strategies, is presented in this paper. Three main strategies, a learning strategy, a niching strategy, and a mutation strategy, are incorporated into the proposed MBSA algorithm. Learning the best individual in current generation and the best position achieved so far is used to improve the convergence speed. Niching and mutation strategies are used to improve the diversity of the MBSA. Finally, some benchmark functions and three chaotic time series prediction problems based on neural networks are simulated to test the effectiveness of MBSA, and the results are compared with those obtained using some other evolutionary algorithms (EAs). The simulation results indicate that the MBSA outperforms other EAs for most functions and chaotic time series.

A Novel Fractional Gradient-Based Learning Algorithm for Recurrent Neural Networks

In this research, we propose a novel algorithm for learning of the recurrent neural networks called as the fractional back-propagation through time (FBPTT). Considering the potential of the fractional calculus, we propose to use the fractional calculus-based gradient descent method to derive the FBPTT algorithm. The proposed FBPTT method is shown to outperform the conventional back-propagation through time algorithm on three major problems of estimation namely nonlinear system identification, pattern classification and Mackey–Glass chaotic time series prediction.

Multimodality Prediction of Chaotic Time Series with Sparse Hard-Cut EM Learning of the Gaussian Process Mixture Model**Supported by the National Natural Science Foundation of China under Grant No 60972106, the China Postdoctoral Science Foundation under Grant No 2014M561053, the Humanity and Social Science Foundation of Ministry of Education of China under Grant No 15YJA630108, and the Hebei Province Natural Science

The contribution of this work is twofold: (1) a multimodality prediction method of chaotic time series with the Gaussian process mixture (GPM) model is proposed, which employs a divide and conquer strategy. It automatically divides the chaotic time series into multiple modalities with different extrinsic patterns and intrinsic characteristics, and thus can more precisely fit the chaotic time series. (2) An effective sparse hard-cut expectation maximization (SHC-EM) learning algorithm for the GPM model is proposed to improve the prediction performance. SHC-EM replaces a large learning sample set with fewer pseudo inputs, accelerating model learning based on these pseudo inputs. Experiments on Lorenz and Chua time series demonstrate that the proposed method yields not only accurate multimodality prediction, but also the prediction confidence interval. SHC-EM outperforms the traditional variational learning in terms of both prediction accuracy and speed. In addition, SHC-EM is more robust and insusceptible to noise than variational learning.

Multivariate Chaotic Time Series Prediction Based on ELM–PLSR and Hybrid Variable Selection Algorithm

In this paper, a novel method (Hybrid–ELM–PLSR) is proposed based on hybrid variable selection algorithm and improved extreme learning machine (ELM) for multivariate chaotic time series prediction. The hybrid variable selection algorithm combines the advantages of filter and wrapper, effectively balancing the calculation speed and prediction accuracy. Moreover, for ELM, multicollinearity, which can result in ill-condition, is always existent among the hidden layer output matrix. And the optimal number of hidden nodes is also difficult to be determined. Therefore,in order to overcome these problems, an improved ELM (ELM–PLSR) is proposed based on partial least square regression (PLSR). It can effectively enhance the stability performance and prediction performance of ELM. Hybrid–ELM–PLSR can be divided into three stages. At first, filter is used to rearrange the input variables through the correlations with desired variables. Then wrapper is used to select the optimal variable subset through evaluating the prediction performance of different subsets. Finally, ELM–PLSR is used to build the prediction model. The simulation experiment results based on San Francisco river runoff dataset demonstrate that the proposed method is effective for multivariate chaotic time series. And the prediction accuracy and reliability are higher than other methods.

Co-evolutionary multi-task learning with predictive recurrence for multi-step chaotic time series prediction

Multi-task learning employs a shared representation of knowledge for learning several instances of the same problem. Multi-step time series problem is one of the most challenging problems for machine learning methods. The performance of a prediction model face challenges for higher prediction horizons due to the accumulation of errors. Cooperative coevolution employs in a divide and conquer approach for training neural networks and has been very promising for single step ahead time series prediction. Recently, co-evolutionary multi-task learning has been proposed for dynamic time series prediction. In this paper, we adapt co-evolutionary multi-task learning for multi-step prediction where predictive recurrence is developed to feature knowledge from previous states for future prediction horizon. The goal of the paper is to present a network architecture with predictive recurrence which is capable of multi-step prediction through a form of multi-task learning. We employ cooperative neuro-evolution and an evolutionary algorithm as baselines for comparison. The results show that the proposed method provides the best generalization performance in most cases. Comparison of results with the literature has shown to be promising which motivates further application of the approach for related real-world problems.

Chaotic Time Series Prediction Using a Photonic Reservoir Computer with Output Feedback

Reservoir Computing is a bio-inspired computing paradigm for processing time dependent signals (Jaeger andHaas 2004; Maass, Natschläger, and Markram 2002). It canbe easily implemented in hardware. The performance ofthese analogue devices matches digital algorithms on a series of benchmark tasks (see e.g. (Soriano et al. 2015) fora review). Their capacities could be extended by feedingthe output signal back into the reservoir, which would allow them to be applied to various signal generation tasks(Antonik et al. 2016b). In practice, this requires a high-speed readout layer for real-time output computation. Herewe achieve this by means of a field-programmable gate array (FPGA), and demonstrate the first photonic reservoircomputer with output feedback. We test our setup on theMackey-Glass chaotic time series generation task and obtain interesting prediction horizons, comparable to numerical simulations, with ample room for further improvement.Our work thus demonstrates the potential offered by the output feedback and opens a new area of novel applications forphotonic reservoir computing.