Nonlinear Time Series Prediction Research Articles

Boosting reservoir computer performance with multiple delays.

Time delays play a significant role in dynamical systems, as they affect their transient behavior and the dimensionality of their attractors. The number, values, and spacing of these time delays influences the eigenvalues of a nonlinear delay-differential system at its fixed point. Here we explore a multidelay system as the core computational element of a reservoir computer making predictions on its input in the usual regime close to fixed point instability. Variations in the number and separation of time delays are first examined to determine the effect of such parameters of the delay distribution on the effectiveness of time-delay reservoirs for nonlinear time series prediction. We demonstrate computationally that an optoelectronic device with multiple different delays can improve the mapping of scalar input into higher-dimensional dynamics, and thus its memory and prediction capabilities for input time series generated by low- and high-dimensional dynamical systems. In particular, this enhances the suitability of such reservoir computers for predicting input data with temporal correlations. Additionally, we highlight the pronounced harmful resonance condition for reservoir computing when using an electro-optic oscillator model with multiple delays. We illustrate that the resonance point may shift depending on the task at hand, such as cross prediction or multistep ahead prediction, in both single delay and multiple delay cases.

Non-linear Time Series Prediction using Improved CEEMDAN, SVD and LSTM

This study recommends a new time series forecasting model, namely ICEEMDAN - SVD - LSTM model, which coalesces Improved Complete Ensemble EMD with Adaptive Noise, Singular Value Decomposition and Long Short Term Memory network. It can be applied to analyse Non-linear and non-stationary data. The framework of this model is comprised of three levels, namely ICEEMDAN level, SVD level and LSTM level. The first level utilized ICEEMDAN to break up the series into some IMF components along with a residue. The SVD in the second level accounts for de-noising of every IMF component and residue. LSTM forecasts all the resultant IMF components and residue in third level. To obtain the forecasted values of the original data, the predictions of all IMF components and residue are added. The proposed model is contrasted with other extant ones, namely LSTM model, EMD - LSTM model, EEMD - LSTM model, CEEMDAN - LSTM model, EEMD - SVD - LSTM model, ICEEMDAN - LSTM model and CEEMDAN - SVD - LSTM model. The comparison bears witness to the potential of the recommended model over the traditional models.

A Koopman–Takens Theorem: Linear Least Squares Prediction of Nonlinear Time Series

The least squares linear filter, also called the Wiener filter, is a popular tool to predict the next element(s) of time series by linear combination of time-delayed observations. We consider observation sequences of deterministic dynamics, and ask: Which pairs of observation function and dynamics are predictable? If one allows for nonlinear mappings of time-delayed observations, then Takens’ well-known theorem implies that a set of pairs, large in a specific topological sense, exists for which an exact prediction is possible. We show that a similar statement applies for the linear least squares filter in the infinite-delay limit, by considering the forecast problem for invertible measure-preserving maps and the Koopman operator on square-integrable functions.

A data-driven implicit deep adaptive neuro-fuzzy inference system capable of manifold learning for function approximation

Fuzzy Neural Networks (FNN) have the ability of decision-making based on constructing semi-ellipsoidal clusters in the input space as the antecedent parts of their fuzzy rules. To determine the output value for each input instance, FNNs consider its membership degree to different sub-regions of the input space. However, forming such meaningful sub-regions is not possible in all applications due to the nonlinear interactions among input variables and their low information gain. Indeed, the samples could be distributed on a manifold in the input space. Therefore, to cover the input space, we need lots of rules, each representing a small region of input space. This issue decreases the generalization ability of the model along with its explainability. Consequently, to efficiently form fuzzy rules, first, it is necessary to unfold the manifold by mapping the samples to an appropriate embedding space. Next, the fuzzy rules in the form of semi-ellipsoidal regions should be constructed in this extracted feature space. Deep Fuzzy Neural Networks address this problem by representation learning through stacking multiple cascade mapping layers. In this paper, we propose a novel approach for nonlinear function approximation and time-series prediction problems, based on using the kernel trick to implicitly learn the mapping function to the new feature space. Moreover, to initialize the fuzzy rules, a KNN-based method using the kernel trick is proposed. A hierarchical Levenberg–Marquardt approach is applied to learn the model’s parameters. The performance and structure of the proposed method are studied and compared with some other relevant methods in synthetic and real-world benchmarks. Based on these experiments, the proposed method has the best performance with the most parsimonious architecture. According to these experiments, the test RMSE of the proposed method is 0.002 for Mc-Glass chaotic time-series prediction, 0.015 for a Nonlinear dynamic system identification, 0.0345 for Box–Jenkins nonlinear system identification, 0.0609 for Fuel consumption prediction of automobiles, 10.24 for Sydney stock price tracking, and 0.595 for California housing.

Stability Analysis of Deep Belief Network: Based SD-AR Model for Nonlinear Time Series

As for nonlinear time series prediction, many different kinds of varying-coefficient models have been proposed and analysised in recent years. A kind of varying functional-coefficient autoregressive model, called the deep belief network-based state-dependent autoregressive (DBN-AR) model is considered in this paper. The stability conditions and existing conditions of limit cycle of the DBN-AR model are also studied. An especial designed parameter estimation method is used to identify the DBN-AR model. The DBN-AR model is used to predict the famous Canadian lynx data and Henon chaotic series, the prediction capability of the DBN-AR model is compared with other prediction models, the experimental results show that the DBN-AR model obtains better prediction accuracy.

Control Method of Lower Limb Rehabilitation Robot Based on Nonlinear Time Series Prediction Model and Sensor Technology

Control Method of Lower Limb Rehabilitation Robot Based on Nonlinear Time Series Prediction Model and Sensor Technology

Optimizing quantum noise-induced reservoir computing for nonlinear and chaotic time series prediction

Quantum reservoir computing is strongly emerging for sequential and time series data prediction in quantum machine learning. We make advancements to the quantum noise-induced reservoir, in which reservoir noise is used as a resource to generate expressive, nonlinear signals that are efficiently learned with a single linear output layer. We address the need for quantum reservoir tuning with a novel and generally applicable approach to quantum circuit parameterization, in which tunable noise models are programmed to the quantum reservoir circuit to be fully controlled for effective optimization. Our systematic approach also involves reductions in quantum reservoir circuits in the number of qubits and entanglement scheme complexity. We show that with only a single noise model and small memory capacities, excellent simulation results were obtained on nonlinear benchmarks that include the Mackey-Glass system for 100 steps ahead in the challenging chaotic regime.

Nonlinear Time Series Analysis and Prediction of General Aviation Accidents Based on Multi-Timescales

General aviation accidents have complex interactions and influences within them that cannot be simply explained and predicted by linear models. This study is based on chaos theory and uses general aviation accident data to conduct research on different timescales (HM-scale, ET-scale, and EF-scale). First, time series are constructed by excluding seasonal patterns from the statistics of general aviation accidents. Secondly, the chaotic properties of multi-timescale series are determined by the 0–1 test and Lyapunov exponent. Finally, by introducing the sparrow search algorithm and tent chaotic mapping, a CSSA-LSSVM prediction model is proposed. The accident data of the National Transportation Safety Board (NTSB) of the United States in the past 15 years is selected for case analysis. The results show that the phase diagram of the 0–1 test presents Brownian motion characteristics, and the maximum Lyapunov exponents of the three scales are all positive, proving the chaotic characteristics of multi-timescale series. The CSSA-LSSVM prediction model’s testing results illustrate its superiority in time series predicting, and when the timescale declines, the prediction error reduces gradually while the fitting effect strengthens and then decreases. This study uncovers the nonlinear chaotic features of general aviation accidents and demonstrates the significance of multi-timescale research in time series analysis and prediction.

Ensemble learning-based nonlinear time series prediction and dynamic multi-objective optimization of organic rankine cycle (ORC) under actual driving cycle

Complicated road conditions make organic Rankine cycle (ORC) operation characteristics show hysteresis and uncertainty. Under the strong coupling correlation of many operating parameters, how to realize the dynamic optimization of ORC comprehensive performance is the key to obtain practical application potential. Based on ensemble learning mechanism, neural network modeling, ensemble system, unsupervised learning, partial mutual information and optimization algorithm are integrated. This paper presents a nonlinear time series prediction and dynamic multi-objective optimization scheme. The average accuracy increased by at least 59.6%. Taking the thermodynamic performance and environmental impact as optimization objectives, dynamic multi-objective optimization is carried out under road conditions. The optimization scheme can effectively trade off the nonlinear correlation between thermal efficiency and emissions of CO2 equivalent.

Dual-kernel echo state network for nonlinear time series prediction

An echo state network (ESN) is a recurrent neural network (RNN) often applied to nonlinear time series prediction. The traditional ESN randomly generates the weights of the input layer and the reservoir layer and does not change these weights, and generally only learns the weights of the output layer through linear regression, so the training speed is very fast. In this work, we propose a novel kernel echo state network (KESN). In KESN, the random weights of the input layer are removed and a series of gaussian kernels are used to replace the neurons in the input layer. Similar to radial basis function (RBF) neural networks, KESN can use the k-means algorithm to generate the kernel center and estimate the bandwidth of the kernel function. We prove that a KESN has echo state property, which is an important factor of KESN that can normally work. Furthermore, kernel ridge regression (KRR) is used to learn the weights of the output layer instead of a simple linear model. Finally, to obtain the optimal parameters of the model, the tree-structured parzen estimator approach (TPE) is used to optimize the hyperparameters of the model. In a time series prediction experiment, it is proved that KESN is more stable and performs better than the echo state network which randomly generates weights and trains output weights using linear models. We found that the reservoir layer weights are equivalent to a dropout operation, and the KESN is inherently equivalent to a regularized neural network. We call the KRR-based KESN dual-kernel echo state network (DKESN).

Robust kernel adaptive filtering for nonlinear time series prediction

Recently, online learning algorithms in machine learning have been imposed much attention. As a typical family, kernel adaptive filtering algorithms receive particular interest due to their sequential learning-based features. However, the kernel least mean square (KLMS) algorithm is not suitable for nonlinear tasks corrupted by non-Gaussian noise, especially impulsive noise. This is because the derivation of the KLMS algorithm is on the basis of the mean square error (MSE) criterion which only captures information of second-order statistics. In this paper, motivated by tanh function, we develop its generalized variant by introducing a scale factor for better representation capability; then we incorporate kernel adaptive filter with the generalized tanh function to propose a robust sequential learning algorithm. Based on establishing the energy conservation relation, we derive a sufficient condition for ensuring the algorithm convergence. In addition, to perform the steady-state excess mean square error (EMSE) analysis, we use the pre-tuned dictionary strategy to model the unknown nonlinear system in form of a finite-order combination; by Taylor expansion, we arrive at a closed-form solution for predicting the steady-state behavior. To further improve the algorithm performance, we design an optimization scheme for scale factor. Simulations for nonlinear time series prediction show that the designed schemes yield better performance than some state-of-art algorithms. The steady-state EMSE analysis is validated to provide accurate prediction results.

Experimental demonstration of bandwidth enhancement in photonic time delay reservoir computing.

Time delay reservoir computing (TDRC) using semiconductor lasers (SLs) has proven to be a promising photonic analog approach for information processing. One appealing property is that SLs subject to delayed optical feedback and external optical injection, allow for tuning the response bandwidth by changing the level of optical injection. Here we use strong optical injection, thereby expanding the SL's modulation response up to tens of gigahertz. Performing a nonlinear time series prediction task, we demonstrate experimentally that for appropriate operating conditions, our TDRC system can operate with sampling times as small as 11.72 ps, without sacrificing computational performance.

Nonlinear prediction of functional time series

AbstractWe propose a nonlinear prediction (NOP) method for functional time series. Conventional methods for functional time series are mainly based on functional principal component analysis or functional regression models. These approaches rely on the stationary or linear assumption of the functional time series. However, real data sets are often nonstationary, and the temporal dependence between trajectories cannot be captured by linear models. Conventional methods are also hard to analyze multivariate functional time series. To tackle these challenges, the NOP method employs a nonlinear mapping for functional data that can be directly applied to multivariate functions without any preprocessing step. The NOP method constructs feature space with forecast information, hence it provides a better ground for predicting future trajectories. The NOP method avoids calculating covariance functions and enables online estimation and prediction. We examine the finite sample performance of the NOP method with simulation studies that consider linear, nonlinear and nonstationary functional time series. The NOP method shows superior prediction performances in comparison with the conventional methods. Three real applications demonstrate the advantages of the NOP method model in predicting air quality, electricity price and mortality rate.

Interval type-2 fuzzy neural networks with asymmetric MFs based on the twice optimization algorithm for nonlinear system identification

This paper proposes a novel algorithm twice optimization for interval type-2 fuzzy neural networks with asymmetric membership functions (TOIT2FNN-AMF), for nonlinear system identification problems. The proposed TOIT2FNN-AMF uses an asymmetric Gaussian interval type-2 membership function to enhance the network’s ability to describe and solve nonlinear and uncertain problems. The twice optimization algorithm consists of structure learning and parameter learning. Firstly, this paper proposes a multi-strategy adaptive differential evolution (MSADE) algorithm as the first optimization algorithm, which is used to determine the structure and the initial values of the parameters of the TOIT2FNN-AMF. It applies the root mean square error (RMSE) of the TOIT2FNN-AMF as the fitness function to determine the structure (number of rules) and initial parameters of the IT2FNN by searching for the RMSE values under different structures. When the fitness value reaches the minimum, that is, the RMSE value of the TOIT2FNN-AMF, the corresponding number will become the optimal one of fuzzy rules of the TOIT2FNN-AMF. Then, the second optimization algorithm of the TOIT2FNN-AMF turns into a hybrid optimization algorithm composed of an adaptive moment estimation (Adam) algorithm and recursive least squares (RLS) algorithm. Adam is used to optimize the antecedent parameters of TOIT2FNN-AMF rules, so as to maintain rapid convergence without generating oscillation during the training process; RLS is used to optimize the consequent parameters of TOIT2FNN-AMF rules, so that the network parameters can be optimized rapidly. In this way, the problems of excessive parameters to be adjusted and excessive slow convergence of the network can be solved. Finally, this paper evaluates the proposed TOIT2FNN-AMF by testing on problems of nonlinear system identification and chaotic time-series prediction. The simulation results are compared with those of similar methods in the existing literatures, which demonstrates that the proposed TOIT2FNN-AMF model yields a lower RMSE value and a simpler network structure than the other type-2 fuzzy neural networks (T2FNNs).

Multi-Reservoir Echo State Networks with Hodrick–Prescott Filter for nonlinear time-series prediction

The Echo State Network (ESN) is a representative model for reservoir computing, which is capable of high-speed model training for machine learning tasks with time series data. Extended models of the ESN, such as Multi-Reservoir ESNs (MRESNs), have been intensively studied for performance improvement in recent years. In this study, we propose a new model called an HP-MRESN by combining an MRESN with the Hodrick–Prescott (HP) filter for nonlinear time series prediction. The proposed HP-MRESN comprises three basic components: a time series decomposer, a reservoir state extractor, and an ensemble decoder. In the time series decomposer, we recursively leverage the HP filter to decompose original time-series data into multiple trend and cycle components. In the reservoir state extractor, each time series component is fed into a corresponding reservoir-state encoder for generating a reservoir state which is extracted as it is or through the principal component analysis. In the ensemble decoder, the states of multiple reservoirs are collected and processed to produce model outputs. Moreover, we propose a greedy algorithm to automatically find the best model architectures under designated hyperparameters for different prediction tasks. Experimental results on a total of 24 nonlinear time-series prediction tasks with 6 real-world datasets demonstrate that our proposed HP-MRESN not only can outperform some existing representative MRESN models and fully-trained RNN models but also can have relatively low training time. In addition, performance comparisons between the HP-MRESN and related MRESN models with other prepossessing methods show the benefit of time series decompositions using the HP filter. The codes of the proposed method are publicly available on https://github.com/Ziqiang-IRCN/HP-MRESN.

A Novel FRBF‐Type Model for Nonlinear Time Series Prediction

Accurate prediction of time series is complex due to nonlinear characteristics but can play a significant role in practical problem. In this paper, a novel varying‐coefficient hybrid model is proposed to accurately predict the nonlinear time series. A set of fuzzy radial basis function (FRBF) neural networks is used to approximate the varying functional coefficients of the state‐dependent autoregressive model with exogenous variables (SD‐ARX). The obtained model is called the fuzzy radial basis function network‐based autoregressive model with exogenous variables (FRBF‐ARX), which combines the advantages of the FRBF in function approximation and the SD‐ARX model in nonlinear dynamics description. Then, a structured nonlinear parameter optimization method (SNPOM) and the modified multifold cross‐validation criterion are used to estimate the parameters of the proposed varying‐coefficient FRBF‐ARX model. The performances of the FRBF‐ARX model are used to predict the PM2.5 concentration and simulated SISO nonlinear process, respectively, and the performances of the proposed model are also compared and discussed. The experimental results show that the FRBF‐ARX model has better performances of accuracy on nonlinear time series forecasting than that of other models.

Prediction of Mixed-mode I and II effective fracture toughness of several types of concrete using the extreme gradient boosting method and metaheuristic optimization algorithms

Complex sample preparation procedures, the necessity for highly accurate and sensitive instruments, early sample failure, and brittle samples all contribute to the difficulty of measurement of concrete fracture toughness (CFT) in the laboratory. With such limitations on the measuring CFT, it is necessary to develop new and more effective tools. Machine learning (ML) methods are promising tools for non-linear time series prediction. In this study, six extreme gradient boosting (XGBoost)-based meta-heuristic algorithms, including standard XGBoost, XGBoost-particle swarm optimization (PSO), XGBoost-imperialist competitive algorithm (ICA), XGBoost-grey wolf optimization (GWO), XGBoost-shuffled frog leaping algorithm (SFLA), and XGBoost-genetic algorithm (GA) were developed to predict the effective fracture toughness (K_eff) of concrete, utilizing 560 datasets obtained from the central straight notched Brazilian disc (CSNBD) test. Additives of micro silica and powdered stone, which are the most widely used building materials were used in the concrete samples to investigate their effect on the physical and mechanical properties. Therefore, 10 different materials made from a combination of concrete and different percentages of additives, were used in the experiment. The datasets included six input parameters effective on the K_eff, including concrete type (ST), diameter (D), thickness (t), length (L), force (F), and crack angle (α). The XGBoost-based models' performances were compared with those of each other and with four other ML methods of Gaussian process regression (GPR), decision trees (DT), support vector regression (SVR), and K-nearest neighbors (KNN). The ML models’ performance in the prediction of K_eff for both holdout and 5-fold methods from high to low is XGBoost-PSO, XGBoost-ICA, XGBoost-GWO, XGBoost-SFLA, XGBoost-GA, XGBoost, GPR, SVR, DT, and KNN. Considering the datasets utilized in this research, the feature selection method's sensitivity analysis revealed that, except for the parameter D, all other parameters considered in this study, significantly affect the K_eff. By developing the meta-heuristic methods, the ability of the XGBoost-based methods was significantly improved. Finally, this article recommends using the XGBoost-PSO hybrid model to predict the K_eff. This work’s significance is that it allows geotechnical engineers to accurately estimate the K_eff of different types of concrete. In this way, the high time and cost required for the CSNBD test can be eliminated.

Modified echo state network for prediction of nonlinear chaotic time series

Modified echo state network for prediction of nonlinear chaotic time series

Reservoir computing on a silicon platform with a ferroelectric field-effect transistor

Reservoir computing offers efficient processing of time-series data with exceptionally low training cost for real-time computing in edge devices where energy and hardware resources are limited. Here, we report reservoir computing hardware based on a ferroelectric field-effect transistor (FeFET) consisting of silicon and ferroelectric hafnium zirconium oxide. The rich dynamics originating from the ferroelectric polarization dynamics and polarization-charge coupling are the keys leading to the essential properties for reservoir computing: the short-term memory and high-dimensional nonlinear transform function. We demonstrate that an FeFET-based reservoir computing system can successfully solve computational tasks on time-series data processing including nonlinear time series prediction after training with simple regression. Due to the FeFET’s high feasibility of implementation on the silicon platform, the systems have flexibility in both device- and circuit-level designs, and have a high potential for on-chip integration with existing computing technologies towards the realization of advanced intelligent systems.

Reducing echo state network size with controllability matrices.

Echo state networks are a fast training variant of recurrent neural networks excelling at approximating nonlinear dynamical systems and time series prediction. These machine learning models act as nonlinear fading memory filters. While these models benefit from quick training and low complexity, computation demands from a large reservoir matrix are a bottleneck. Using control theory, a reduced size replacement reservoir matrix is found. Starting from a large, task-effective reservoir matrix, we form a controllability matrix whose rank indicates the active sub-manifold and candidate replacement reservoir size. Resulting time speed-ups and reduced memory usage come with minimal error increase to chaotic climate reconstruction or short term prediction. Experiments are performed on simple time series signals and the Lorenz-1963 and Mackey-Glass complex chaotic signals. Observing low error models shows variation of active rank and memory along a sequence of predictions.