Time Series Prediction Research Articles

Post-processing methods for delay embedding and feature scaling of reservoir computers

Reservoir computing is a machine learning method that is well-suited for complex time series prediction tasks. Both delay embedding and the projection of input data into a higher-dimensional space play important roles in enabling accurate predictions. We establish simple post-processing methods that train on past node states at uniformly or randomly-delayed timeshifts. These methods improve reservoir computer prediction performance through increased feature dimension and/or better delay embedding. Here we introduce the multi-random-timeshifting method that randomly recalls previous states of reservoir nodes. The use of multi-random-timeshifting allows for smaller reservoirs while maintaining large feature dimensions, is computationally cheap to optimise, and is our preferred post-processing method. For experimentalists, all our post-processing methods can be translated to readout data sampled from physical reservoirs, which we demonstrate using readout data from an experimentally-realised laser reservoir system.

Time-series prediction of steel corrosion in concrete using recurrent neural networks

ABSTRACT Steel corrosion is a time-series problem where the current corrosion state is influenced by past corrosion and environmental factors. Most models incorporate time as a variable but fail to comprehensively consider past states and spatial-temporal factors. This study collects continuously monitored corrosion data from the literature and develops time-variant corrosion rate prediction models using traditional machine learning (TML) and recurrent neural networks (RNN). The TML model captures temporal variations, while the RNN model incorporates past and current corrosion factors, focusing on different lookback lengths. Empirical and electrochemical models are also compared to evaluate efficacy. The results indicate superior predictive capabilities for bothTML and RNN models. The TML model yields an RMSE of 0.26 μA/cm2 and a MAPE of 11.27%, while the RNN model achieves 0.22 μA/cm2 and 8.15%. In contrast, empirical and electrochemical models yield suboptimal RMSE (1.48 μA/cm2, 0.53 μA/cm2) and MAPE (77.31%, 35.05%). Analysis of Simple-RNN and LSTM models, reveals optimal performance when considering corrosion factors from the preceding two-time intervals. These results underscore the importance of incorporating past corrosion factors and demonstrate the enhanced efficacy of data-driven approaches, particularly LSTM, over traditional models.

A Focused Review on Upper and Lower Limb Joint Torque Estimation via Neural Networks

Joint torque estimation is an essential aspect of the control architecture in assistive devices for rehabilitation and aiding movement impairments. Healthy adult torque trajectories serve as a baseline for controllers to determine the level of assistance required by patients, evaluate impaired motion, understand biomechanics, and design treatment plans. Currently, methods of torque estimation include inverse dynamics using gold standard motion capture systems, generic mathematical models based on joint torque-angle relationships, neuromusculoskeletal modelling using surface electromyography, and neural networks. As such, this review provides a focused overview of the recent and existing neural networks tailored for upper and lower limb joint torque estimation. Dataset preparation, data preprocessing, and evaluation metrics are presented along with a detailed description of the developed networks, which are classified by model architecture. It includes artificial neural networks (ANNs), convolution neural networks (CNNs), long short-term memory (LSTM) networks, and hybrid and alternate architectures such as wavelet or explainable convolution (XCM). The performance, benefits, and limitations of the models are discussed, highlighting CNNs and LSTMs as the current optimal models for time series prediction of joint torque. This is due to their ability to capture spatial and temporal dependencies in the data. Additionally, joint kinematics such as angles, angular velocities, and accelerations are considered optimal input parameters due to their ease of measurement using wearable sensors and integration with wearable assistive technology.

Hybrid recurrent neural network using custom activation function for COVID-19 short term time series prediction

Introduction: The nonlinear behaviour of activation functions is vital in Artificial Neural Networks (ANNs) for exploring the complex relationship between the input and output features. However, these are probably going to encounter vanishing gradient problems due to small gradients that lead to training instability, expensive exponent operations, and slow convergence. Objectives: The primary objective of this study is to develop Taylor expansion of the second order to realize the hyperbolic tangent and sigmoid functions. In particular, long short term memory network make extensive use of these functions as well as gating mechanism to control the flow of information and gradients. Both the custom functions can reduce the vanishing gradient issues in recurrent neural networks. Methods: Taylor expansion hyperbolic tangent and sigmoid activation functions based parallel heterogeneous Long Short Term Memory Network integrated with Bayesian hyperparameter Optimization is being proposed for coronavirus multi step time series prediction. a Min-Max Normalization is applied, which produces scaled data in the range (0, 1). The normalized dataset is partitioned into training and testing datasets, with 80% and 20%, respectively. Furthermore, both train and test datasets are prepared as input and target series using a window size of 5-7.The further proposed model is tuned with key hyperparameters such as the number of neurons, learning rate, dropout, and type of optimizer. The remaining model parameters are epochs, batch size, and loss, which are 200, 32, and mean square error, respectively. Results: The proposed model efficacy is evaluated on coronavirus daily cumulative cases, cumulative deaths, daily new cases, and total recovery cases in India. The Analysis reveals that the current model achieves remarkable performance in terms of Mean Absolute Percentage Error (MAPE), Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Coefficient of determination (R2 Score) when compared to existing models. Conclusions: The study reveals that the proposed framework with the Taylor approximation activation function produces more consistency in prediction than the default activation functions, including Tanh and sigmoid. In spite of that, gradients of Taylor Tanh and sigmoid activation function traits indicate a decline in the possibility of vanishing issue.

An adaptive ensemble learning capacity prediction method with dual-modal data in lithium-ion battery manufacturing

ABSTRACT To measure the capacity of lithium-ion batteries in manufacturing, it is necessary to go through the formation, standing, and grading processes. The existing data-driven capacity prediction methods can only reduce a portion of the grading process and do not comprehensively use both time series data and statistical feature data. In response to the above problems, this paper proposes an adaptive ensemble learning method, which does not go through the grading process and only uses the statistical features and time series of dual-modal data in the formation process to predict the capacity. The LightGBM algorithm is used to predict statistical feature data, while the multi-channel GRU and multi-head self-attention network are used to predict time series data. Then, the two parts are fused through adaptive ensemble learning to estimate the capacity. Furthermore, the proposed method could be flexibly applied to the prediction of single-modal dirty data as required. The method is verified using actual industrial production line data. The experimental results show that the MAE and MAPE of the test set are 0.948 Ah and 0.324%, respectively. For single-modal dirty data (accounting for about 4%), MAPE of its prediction results is less than 0.65%.

Quantum-inspired neural networks for time-series air pollution prediction and control of the most polluted region in the world

Quantum-inspired neural networks for time-series air pollution prediction and control of the most polluted region in the world

Forecasting Agricultural Trade Based on TCN-LightGBM Models: A Data-Driven Decision

Facing the increasing complexity and dynamic fluctuations of the global agricultural trade market, accurate forecasting plays a key role in supporting agricultural policy formulation, stabilising the market and optimising resource allocation. In order to increase the precision and stability of agricultural trade predictions, this research suggests a hybrid model built on a temporal convolution network (TCN) and a lightweight gradient boosting tree (LightGBM). The TCN module effectively captures the long-term dependence characteristics of time series data through dilated convolution, which improves the model’s ability to identify seasonal and periodic trends. The LightGBM module, on the other hand, makes use of the characteristics of gradient boosting decision trees and excels at efficiently handling nonlinear relationships and avoiding overfitting. Experimental results show that the TCN-LightGBM model outperforms traditional models in terms of mean square error (MSE), mean absolute error (MAE) and prediction accuracy. Specifically, compared with ARIMA, LSTM, TCN alone or LightGBM alone, TCN-LightGBM achieves a prediction accuracy of 91.3% on the test data, with MSE and MAE of 0.021 and 0.115 respectively, significantly improving prediction accuracy and stability. In addition, parameter sensitivity analysis shows that the TCN-LightGBM model maintains a highly consistent prediction trend under different parameter configurations, which verifies the robustness of the model and its practical application value. This study provides a data-driven decision support tool with high accuracy and strong stability, providing a new solution for agricultural trade forecasting and other complex time series prediction tasks.

Principled neuromorphic reservoir computing

Reservoir computing advances the intriguing idea that a nonlinear recurrent neural circuit—the reservoir—can encode spatio-temporal input signals to enable efficient ways to perform tasks like classification or regression. However, recently the idea of a monolithic reservoir network that simultaneously buffers input signals and expands them into nonlinear features has been challenged. A representation scheme in which memory buffer and expansion into higher-order polynomial features can be configured separately has been shown to significantly outperform traditional reservoir computing in prediction of multivariate time-series. Here we propose a configurable neuromorphic representation scheme that provides competitive performance on prediction, but with significantly better scaling properties than directly materializing higher-order features as in prior work. Our approach combines the use of randomized representations from traditional reservoir computing with mathematical principles for approximating polynomial kernels via such representations. While the memory buffer can be realized with standard reservoir networks, computing higher-order features requires networks of ‘Sigma-Pi’ neurons, i.e., neurons that enable both summation as well as multiplication of inputs. Finally, we provide an implementation of the memory buffer and Sigma-Pi networks on Loihi 2, an existing neuromorphic hardware platform.

Cooperative co-evolution deep echo state network for time series prediction

Abstract The deep echo state network demonstrates strong predictive performance in time series tasks, but inter-layer weight optimization and reservoir structure design are still challenging tasks. To address this, we propose the cooperative co-evolutionary deep echo state network (CCEDESN). Firstly, inspired by the hierarchical structure of the brain, we introduce a modular hierarchical design. Each layer of the reservoir contains multiple sub-reservoirs, with central neurons in each sub-reservoir interacting with those in other sub-reservoirs to enhance information processing. Secondly, an advanced cooperative co-evolutionary algorithm is proposed to simultaneously optimize the layer connection weights and hyperparameters of the deep echo state network. We evaluate the CCEDESN model on the Mackey-Glass system, sunspot data, and Apple stock opening prices. The experimental results show that the CCEDESN model achieves superior prediction accuracy compared to other models.

Machine Learning-Based Anomaly Prediction for Proactive Monitoring in Data Centers: A Case Study on INFN-CNAF

Anomaly prediction in time series is crucial for ensuring the stability and security of data centers, especially in scientific contexts such as INFN-CNAF, the National Center for Research and Development in Information and Communication Technology of the National Institute for Nuclear Physics. At INFN-CNAF, large volumes of heterogeneous data critical to international experiments are managed using dedicated monitoring systems. To ensure continuous availability, artificial intelligence solutions are being explored to detect anomalies and predict potential failures proactively. This work presents a machine learning-based approach for automatic anomaly prediction in the operational metrics of INFN-CNAF’s WebDav service. We evaluate several methods, including Long Short-Term Memory, Random Forest, and various neural networks, assessing their Accuracy and sensitivity in distinguishing normal from anomalous behaviors. The results demonstrate the effectiveness of these methods, not only in predicting anomalies but also in pinpointing critical areas within monitored metrics. This contributes to more proactive IT resource monitoring and enhances data center management efficiency.

Harnessing spatiotemporal transformation in magnetic domains for nonvolatile physical reservoir computing.

Combining physics with computational models is increasingly recognized for enhancing the performance and energy efficiency in neural networks. Physical reservoir computing uses material dynamics of physical substrates for temporal data processing. Despite the ease of training, building an efficient reservoir remains challenging. Here, we explore beyond the conventional delay-based reservoirs by exploiting the spatiotemporal transformation in all-electric spintronic devices. Our nonvolatile spintronic reservoir effectively transforms the history dependence of reservoir states to the path dependence of domains. We configure devices triggered by different pulse widths as neurons, creating a reservoir featured by strong nonlinearity and rich interconnections. Using a small reservoir of merely 14 physical nodes, we achieved a high recognition rate of 0.903 in written digit recognition and a low error rate of 0.076 in Mackey-Glass time series prediction on a proof-of-concept printed circuit board. This work presents a promising route of nonvolatile physical reservoir computing, which is adaptable to the larger memristor family and broader physical neural networks.

Modeling and Evaluation of Attention Mechanism Neural Network Based on Industrial Time Series Data

Chemical process control systems are complex, and modeling the controlled object is the first task in automatic control and optimal design. Most chemical process modeling experiments require test signals to be applied to the process, which may lead to production interruptions or cause safety accidents. Therefore, this paper proposes an improved transformer model based on a self-attention mechanism for modeling industrial processes. Then, an evaluation mechanism based on root mean square error (RMSE) and Kullback–Leibler divergence (KLD) metrics is designed to obtain more appropriate model parameters. The Variational Auto-Encoder (VAE) network is used to compute the associated KLD. Finally, a real nonlinear dynamic process in the petrochemical industry is modeled and evaluated using the proposed methodology to predict the time series data of the process. This study demonstrates the validity of the proposed transformer model and illustrates the versatility of using an integrated modeling, evaluation, and prediction scheme for nonlinear dynamic processes in process industries. The scheme is of great importance for the field of industrial soft measurements as well as for deep learning-based time series prediction. In addition, the issue of a suitable time domain for the prediction is discussed.

Exploring the role of cardiac activity in forecasting cognitive fatigue with machine learning

Fatigue poses significant risks to safety, productivity, and overall well-being. Traditional statistical methods have been employed for inferring fatigue-related patterns; however, there is a need for interpretable machine learning approaches that integrate time series prediction capabilities to develop intelligent health systems for fatigue management. This study explored forecasting perception and performance scores associated with fatigue using cardiac activity and heart rate variability (HRV), employing both generalized and personalized models with a 10-minute forecast interval during a cognitively fatiguing task. Participants underwent a two-hour working memory task while providing subjective fatigue responses every 10 min, serving as perception labels, and their performance within each 10-minute interval was quantified as their performance labels. The results revealed that performance-labeled models generated lower mean absolute errors compared to perception-labeled models, while the Gradient Boosting Regression algorithm achieved the lowest mean absolute error in forecasting performance scores due to fatigue for both generalized and personalized models. Sample entropy, ratio of Standard Deviation-Poincaré, the proportion of peak-to-peak intervals over 50 ms, coefficient of variation of peak-to-peak intervals, and low frequency were the most important features for predicting performance. These findings offer the potential to forecast performance decline resulting from fatigue in working memory tasks, facilitating the implementation of fatigue mitigation interventions to reduce injury risks and performance impairments. The integration of interpretable machine learning methods with time series forecasting provides valuable insights for developing intelligent systems that proactively manage fatigue and optimize performance across various domains.

Time series predictions in unmonitored sites: a survey of machine learning techniques in water resources

Abstract Prediction of dynamic environmental variables in unmonitored sites remains a long-standing challenge for water resources science. The majority of the world’s freshwater resources have inadequate monitoring of critical environmental variables needed for management. Yet, the need to have widespread predictions of hydrological variables such as river flow and water quality has become increasingly urgent due to climate and land use change over the past decades, and their associated impacts on water resources. Modern machine learning methods increasingly outperform their process-based and empirical model counterparts for hydrologic time series prediction with their ability to extract information from large, diverse data sets. We review relevant state-of-the art applications of machine learning for streamflow, water quality, and other water resources prediction and discuss opportunities to improve the use of machine learning with emerging methods for incorporating watershed characteristics and process knowledge into classical, deep learning, and transfer learning methodologies. The analysis here suggests most prior efforts have been focused on deep learning frameworks built on many sites for predictions at daily time scales in the United States, but that comparisons between different classes of machine learning methods are few and inadequate. We identify several open questions for time series predictions in unmonitored sites that include incorporating dynamic inputs and site characteristics, mechanistic understanding and spatial context, and explainable AI techniques in modern machine learning frameworks.

Time series analysis and prediction of climate variables of Southern Java waters using support vector regression

Time series analysis and prediction of climate variables of Southern Java waters using support vector regression

Real-time adaptation for time-series signal prediction using label-aware neural processes

Real-time adaptation for time-series signal prediction using label-aware neural processes

A multi-modal image encoding and self-attention-based transformer framework with sentiment analysis for financial time series prediction

A multi-modal image encoding and self-attention-based transformer framework with sentiment analysis for financial time series prediction

LSTM-Based Time Series Prediction Model: A Case Study with YFinance Stock Data

The goal of this study is to anticipate the time series of stock data that YFinance provides using a Long Short-Term Memory (LSTM) model, with a particular emphasis on the closing prices and daily returns of Apple Inc. (AAPL). The historical closing price data from January 1, 2010, to September 20, 2021, was used as the training set, while the data from September 21, 2021, to August 22, 2024, was employed as the validation set to test the model’s predictive capability. The experimental results demonstrate that the LSTM architecture performs excellently in handling data with long-term dependencies and trends, attaining a root mean square error (RMSE) of 5.2129 and a coefficient of determination (R2) of 0.94, thus accurately forecasting the stock price movements of Apple Inc. However, the model exhibits poor performance in predicting high-frequency fluctuations, with an R2 of only -0.1, indicating a weak ability to capture high-frequency volatility.

A Dynamic Interval Auto-Scaling Optimization Method Based on Informer Time Series Prediction

A Dynamic Interval Auto-Scaling Optimization Method Based on Informer Time Series Prediction

Temporal fusion point-interval forecasting: A comprehensive approach for financial time series prediction

Temporal fusion point-interval forecasting: A comprehensive approach for financial time series prediction