Networks For Time Series Prediction Research Articles

Multi-scale deep echo state network for time series prediction

Multi-scale deep echo state network for time series prediction

Fractional-integer-order echo state network for time series prediction

In this paper, a new echo state network with fractional-order reservoir and integer-order reservoir in series configuration, called fractional-integer-order ESN (FIO-ESN), is proposed for time series prediction. Firstly, considering the infinite memory of fractional-order reservoir, the feature information of input signals will be amplified through the fractional-order reservoir, and then the magnified feature information can be extracted twice by using the integer-order reservoir with very large input weights. Secondly, the magnitude of the fractional-order reservoir state is increased through the integer-order reservoir, and then the output weight can be computed in a reasonable range. Thirdly, in order to realize the stable application of the FIO-ESN, a sufficient stability criterion for the FIO-ESN is given by using an LMI approach. Fourthly, in order to reduce the dependence of the prediction accuracy of the FIO-ESN on the fractional-integer-order reservoir parameters, an optimization algorithm based on gradient descent is given. Finally, two numerical simulation examples and one real-world example are used for demonstrating the feasibility of stability criterion and the learning performance of the FIO-ESN.

EgPDE-Net: Building Continuous Neural Networks for Time Series Prediction With Exogenous Variables.

While exogenous variables have a major impact on performance improvement in time series analysis, interseries correlation and time dependence among them are rarely considered in the present continuous methods. The dynamical systems of multivariate time series could be modeled with complex unknown partial differential equations (PDEs) which play a prominent role in many disciplines of science and engineering. In this article, we propose a continuous-time model for arbitrary-step prediction to learn an unknown PDE system in multivariate time series whose governing equations are parameterized by self-attention and gated recurrent neural networks. The proposed model, exogenous-guided PDE network (EgPDE-Net), takes account of the relationships among the exogenous variables and their effects on the target series. Importantly, the model can be reduced into a regularized ordinary differential equation (ODE) problem with specially designed regularization guidance, which makes the PDE problem tractable to obtain numerical solutions and feasible to predict multiple future values of the target series at arbitrary time points. Extensive experiments demonstrate that our proposed model could achieve competitive accuracy over strong baselines: on average, it outperforms the best baseline by reducing 9.85% on RMSE and 13.98% on MAE for arbitrary-step prediction.

Memory augmented echo state network for time series prediction

Memory augmented echo state network for time series prediction

Multiscale convolutional neural-based transformer network for time series prediction

Multiscale convolutional neural-based transformer network for time series prediction

Remaining Useful Life Prediction of Rolling Bearings Based on Multi-scale Permutation Entropy and ISSA-LSTM.

The performance of bearings plays a pivotal role in determining the dependability and security of rotating machinery. In intricate systems demanding exceptional reliability and safety, the ability to accurately forecast fault occurrences during operation holds profound significance. Such predictions serve as invaluable guides for crafting well-considered reliability strategies and executing maintenance practices aimed at enhancing reliability. In the real operational life of bearings, fault information often gets submerged within the noise. Furthermore, employing Long Short-Term Memory (LSTM) neural networks for time series prediction necessitates the configuration of appropriate parameters. Manual parameter selection is often a time-consuming process and demands substantial prior knowledge. In order to ensure the reliability of bearing operation, this article investigates the application of three advanced techniques-Maximum Correlation Kurtosis Deconvolution (MCKD), Multi-Scale Permutation Entropy (MPE), and Long Short-Term Memory (LSTM) recurrent neural networks-for the prediction of the remaining useful life (RUL) of rolling bearings. The improved sparrow search algorithm (ISSA) is employed for configuring parameters in the Long Short-Term Memory (LSTM) network. Each technique's principles, methodologies, and applications are comprehensively reviewed, offering insights into their respective strengths and limitations. Case studies and experimental evaluations are presented to assess their performance in RUL prediction. Findings reveal that MCKD enhances fault signatures, MPE captures complexity, and LSTM excels in modeling temporal patterns. The root mean square error of the prediction results is 0.007. The fusion of these techniques offers a comprehensive approach to RUL prediction, leveraging their unique attributes for more accurate and reliable predictions.

A Hybrid Model Using PCA and BP Neural Network for Time Series Prediction in Chinese Stock Market with TOPSIS Analysis

The stock price changes rapidly and is highly nonlinear in the financial market. One of the common concerns of many scholars and investors is how to accurately predict the stock price and the trend of rising and falling in a short time. Machine learning and deep learning techniques have found their place in financial institutions thanks to the ability of time series data prediction with high precision. However, the prediction accuracy of these models is still far from satisfactory. Most existing studies use original, single prediction algorithms that cannot overcome inherent limitations. This study proposes a hybrid model using principal component analysis (PCA) and backpropagation (BP) neural networks. The historical records of China Merchants Bank are used for data collection from 2015 to 2021. PCA preprocesses the original data to reduce the dimensionality and is then adopted by the BP neural network to predict the stock closing price of China Merchants Bank. We compare and analyze the PCA–BP model with three training algorithms, and the results indicate that the Bayesian regularization algorithm performs best. Besides, we perform the stock prediction using a traditional exponential smoothing approach. The experiment results show that the predicted stock closing price is close to the actual value, and the mean absolute percentage error can reach 0.0130, which is more significant than the traditional approach. Furthermore, A TOPSIS approach is utilized to evaluate the robustness of the proposed model. Finally, we demonstrate the usability of the designed hybrid model by predicting the stock price of another selected stock.

A robust training of dendritic neuron model neural network for time series prediction

A robust training of dendritic neuron model neural network for time series prediction

An Empirical Mode Decomposition Fuzzy Forecast Model for Air Quality.

Air quality has a significant influence on people's health. Severe air pollution can cause respiratory diseases, while good air quality is beneficial to physical and mental health. Therefore, the prediction of air quality is very important. Since the concentration data of air pollutants are time series, their time characteristics should be considered in their prediction. However, the traditional neural network for time series prediction is limited by its own structure, which makes it very easy for it to fall into a local optimum during the training process. The empirical mode decomposition fuzzy forecast model for air quality, which is based on the extreme learning machine, is proposed in this paper. Empirical mode decomposition can analyze the changing trend of air quality well and obtain the changing trend of air quality under different time scales. According to the changing trend under different time scales, the extreme learning machine is used for fast training, and the corresponding prediction value is obtained. The adaptive fuzzy inference system is used for fitting to obtain the final air quality prediction result. The experimental results show that our model improves the accuracy of both short-term and long-term prediction by about 30% compared to other models, which indicates the remarkable efficacy of our approach. The research of this paper can provide the government with accurate future air quality information, which can take corresponding control measures in a targeted manner.

Self-organizing pipelined recurrent wavelet neural network for time series prediction

The pipelined recurrent wavelet neural network (PRWNN) with nested structure has been widely used for time series prediction. However, selecting the appropriate network structure for different problems is still an unsolved problem. To solve this issue, the self-organizing PRWNN (SPRWNN) is proposed. Different from the PRWNN with fixed structure, the number of modules and hidden neurons of SPRWNN can be automatically determined during the training process. Firstly, the number of hidden neurons in each module of the SPRWNN is adjusted by the spiking strength (SS) of nodes. Then, the module growth mechanism is designed by measuring the performance of the SPRWNN. Furthermore, the convergence of SPRWNN is proved theoretically. The experimental results show that the SPRWNN can not only automatically adjust the network structure, but also improves the prediction accuracy by about 40% as compared with the PRWNN with fixed structure.

DAFA-BiLSTM: Deep Autoregression Feature Augmented Bidirectional LSTM network for time series prediction

Time series forecasting models that use the past information of exogenous or endogenous sequences to forecast future series play an important role in the real world because most real-world time series datasets are rich in time-dependent information. Most conventional prediction models for time series datasets are time-consuming and fraught with complex limitations because they usually fail to adequately exploit the latent spatial dependence between pairs of variables. As a successful variant of recurrent neural networks, the long short-term memory network (LSTM) has been demonstrated to have stronger nonlinear dynamics to store sequential data than traditional machine learning models. Nevertheless, the common shallow LSTM architecture has limited capacity to fully extract the transient characteristics of long interval sequential datasets. In this study, a novel deep autoregression feature augmented bidirectional LSTM network (DAFA-BiLSTM) is proposed as a new deep BiLSTM architecture for time series prediction. Initially, the input vectors are fed into a vector autoregression (VA) transformation module to represent the time-delayed linear and nonlinear properties of the input signals in an unsupervised way. Then, the learned nonlinear combination vectors of VA are progressively fed into different layers of BiLSTM and the output of the previous BiLSTM module is also concatenated with the time-delayed linear vectors of the VA as an augmented feature to form new additional input signals for the next adjacent BiLSTM layer. Extensive real-world time series applications are addressed to demonstrate the superiority and robustness of the proposed DAFA-BiLSTM. Comparative experimental results and statistical analysis show that the proposed DAFA-BiLSTM has good adaptive performance as well as robustness even in noisy environment.

A PLS-based pruning algorithm for simplified long–short term memory neural network in time series prediction

As an extensively used model for time series prediction, the Long–Short Term Memory (LSTM) neural network suffers from shortcomings such as high computational cost and large memory requirement, due to its complex structure. To address these problems, a PLS-based pruning algorithm is hereby proposed for a simplified LSTM (PSLSTM). First, a hybrid strategy is designed to simplify the internal structure of LSTM, which combines the structure simplification and parameter reduction for gates. Second, partial least squares (PLS) regression coefficients are used as the metric to evaluate the importance of the memory blocks, and the redundant hidden layer size is pruned by merging unimportant blocks with their most correlated ones. The Backpropagation Through Time (BPTT) algorithm is utilized as the learning algorithm to update the network parameters. Finally, several benchmark and practical datasets for time series prediction are used to evaluate the performance of the proposed PSLSTM. The experimental results demonstrate that the PLS-based pruning algorithm can achieve the trade-off between a good generalization ability and a compact network structure. The computational complexity is improved by the simple internal structure as well as the compact hidden layer size, without sacrificing prediction accuracy.

A Novel Broad Echo State Network for Time Series Prediction: Cascade of Mapping Nodes and Optimization of Enhancement Layer

Time series prediction is crucial for advanced control and management of complex systems, while the actual data are usually highly nonlinear and nonstationary. A novel broad echo state network is proposed herein for the prediction problem of complex time series data. Firstly, the framework of the broad echo state network with cascade of mapping nodes (CMBESN) is designed by embedding the echo state network units into the broad learning system. Secondly, the number of enhancement layer nodes of the CMBESN is determined by proposing an incremental algorithm. It can obtain the optimal network structure parameters. Meanwhile, an optimization method is proposed based on the nonstationary statistic metrics to determine the enhancement layer. Finally, experiments are conducted both on the simulated and actual datasets. The results show that the proposed CMBESN and its optimization have good prediction capability for nonstationary time series data.

Self-Attentive Moving Average for Time Series Prediction

Time series prediction has been studied for decades due to its potential in a wide range of applications. As one of the most popular technical indicators, moving average summarizes the overall changing patterns over a past period and is frequently used to predict the future trend of time series. However, traditional moving average indicators are calculated by averaging the time series data with equal or predefined weights, and ignore the subtle difference in the importance of different time steps. Moreover, unchanged data weights will be applied across different time series, regardless of the differences in their inherent characteristics. In addition, the interaction between different dimensions of different indicators is ignored when using the moving averages of different scales to predict future trends. In this paper, we propose a learning-based moving average indicator, called the self-attentive moving average (SAMA). After encoding the input signals of time series based on recurrent neural networks, we introduce the self-attention mechanism to adaptively determine the data weights at different time steps for calculating the moving average. Furthermore, we use multiple self-attention heads to model the SAMA indicators of different scales, and finally combine them through a bilinear fusion network for time series prediction. Extensive experiments on two real-world datasets demonstrate the effectiveness of our approach. The data and codes of our work have been released.

Grey Wolf Optimization–Based Deep Echo State Network for Time Series Prediction

The Echo State Network (ESN) is a unique type of recurrent neural network. It is built atop a reservoir, which is a sparse, random, and enormous hidden infrastructure. ESN has been successful in dealing with a variety of non-linear issues, including prediction and classification. ESN is utilized in a variety of architectures, including the recently proposed Multi-Layer (ML) architecture. Furthermore, Deep Echo State Network (DeepESN) models, which are multi-layer ESN models, have recently been proved to be successful at predicting high-dimensional complicated non-linear processes. The proper configuration of DeepESN architectures and training parameters is a time-consuming and difficult undertaking. To achieve the lowest learning error, a variety of parameters (hidden neurons, input scaling, the number of layers, and spectral radius) are carefully adjusted. However, the optimum training results may not be guaranteed by this haphazardly created work. The grey wolf optimization (GWO) algorithm is introduced in this study to address these concerns. The DeepESN based on GWO (GWODESN) is utilized in trials to forecast time series, and therefore the results are compared with the regular ESN, LSTM, and ELM models. The findings indicate that the planned model performs the best in terms of prediction.

Wavelet Interval Type-2 Takagi-Kang-Sugeno Hybrid Controller for Time-Series Prediction and Chaotic Synchronization

This paper presents a new hybrid neural network for time series prediction and chaotic synchronization. The proposed controller consists of a wavelet interval type-2 Takagi Sugeno Kang fuzzy brain emotional learning controller (WIT2TFBELC), a wavelet interval type-2 Takagi Sugeno Kang fuzzy cerebellar model articulation controller (WIT2TFCMAC), and a robust compensator (RB). The main controller combines the WIT2TFBELC and the WIT2TFCMAC. A TSK fuzzy system is used to create a hybrid structure with the WIT2TFBELC and the WIT2TFCMAC. The TSK fuzzy system can effectively adjust the weighting for the main controller to achieve the time series prediction and chaotic prediction with better tracking response. Morever a robust compensator is used to achieve robust ability of the system. A Lyapunov function was used to establish the adaptive laws and effectively adjust the system parameters online. Finally, two examples of the application of the proposed algorithm are presented to point out the performance of proposed method.

A pseudo-inverse decomposition-based self-organizing modular echo state network for time series prediction

Echo state network (ESN) refers to a popular recurrent neural network with a largely and randomly generated reservoir for its rapid learning ability. However, it is difficult to design a reservoir that matches a specific task. To solve the structure design of the reservoir, a pseudo-inverse decomposition-based self-organizing modular echo state (PDSM-ESN) is proposed. PDSM-ESN is constructed by growing–pruning method, where the error and condition number are used, respectively. Since the self-organizing process may negatively affect the learning speed, the pseudo-inverse decomposition is adopted to improve learning speed, which means the output weights are learned by an iterative incremental method. Meanwhile, to solve the ill-posed problem, the modular sub-reservoirs corresponding to the high condition number are pruned. Simulation results indicate that PDSM-ESN has better prediction performance and run-time complexity compared with the traditional ESN models.

Self-Adaptive Particle Swarm Optimization-Based Echo State Network for Time Series Prediction.

Echo state networks (ESNs), belonging to the family of recurrent neural networks (RNNs), are suitable for addressing complex nonlinear tasks due to their rich dynamic characteristics and easy implementation. The reservoir of the ESN is composed of a large number of sparsely connected neurons with randomly generated weight matrices. How to set the structural parameters of the ESN becomes a difficult problem in practical applications. Traditionally, the design of the parameters of the ESN structure is performed manually. The manual adjustment of the ESN parameters is not convenient since it is an extremely challenging and time-consuming task. This paper proposes an ensemble of five particle swarm optimization (PSO) strategies to design the structure of ESN and then reduce the manual intervention in the design process. An adaptive selection mechanism is used for each particle in the evolution to select a strategy from the strategy candidate pool for evolution. In addition, leaky integration neurons are used as reservoir internal neurons, which are added within the adaptive mechanism for optimization. The root mean squared error (RMSE) is adopted as the evaluation criterion. The experimental results on Mackey-Glass time series benchmark dataset show that the proposed method outperforms other traditional evolutionary methods. Furthermore, experimental results on electrocardiogram dataset show that the proposed method on the ensemble of PSO displays an excellent performance on real-world problems.

DWNet: Dual-Window Deep Neural Network for Time Series Prediction

Multivariate time series prediction is a very important task, which plays a huge role in climate, economy, and other fields. We usually use an Attention-based Encoder-Decoder network to deal with multivariate time series prediction because the attention mechanism makes it easier for the model to focus on the really important attributes. However, the Encoder-Decoder network has the problem that the longer the length of the sequence is, the worse the prediction accuracy is, which means that the Encoder-Decoder network cannot process long series and therefore cannot obtain detailed historical information. In this paper, we propose a dual-window deep neural network (DWNet) to predict time series. The dual-window mechanism allows the model to mine multigranularity dependencies of time series, such as local information obtained from a short sequence and global information obtained from a long sequence. Our model outperforms nine baseline methods in four different datasets.

Definition of neural network model for time series prediction

This article is devoted to solving the issue of forecasting time series in the context of the functionality of project management systems, using the computational resources of neural networks. The article contains the results of analysis of neural network models and conducted experimental studies of the quality of their work, the neural network configuration was selected and adjusted, which allows predicting project execution time with sufficient accuracy. In the process of determining the structure of the neural network for time series prediction, several variants of recurring neural networks and LTSM models with different number of layers and neurons in them were considered, with varying sets of input data. Testing of the quality of neural networks was carried out on a sample of more than 300 thousand tasks for 15 projects accumulated over 8 years. Analysis of the convergence of the data predicted by the neural networks and the real deadlines for completing the tasks made it possible to design the final architecture of the neural network to predict the time of execution of projects. The neural network was developed for further integration into the project management system. The developed neural network was trained and tested. The test results showed that the prediction error in this case was determined within 20 hours. Evaluation of the duration of the task is usually carried out in days, respectively, forecasting by the developed system is carried out up to the day, which can be considered a completely satisfactory result