Autoregressive Prediction Model Research Articles

Energy-efficient neural network training through runtime layer freezing, model quantization, and early stopping

Background:In the last years, neural networks have been massively adopted by industry and research in a wide variety of contexts. Neural network milestones are generally reached by scaling up computation, completely disregarding the carbon footprint required for the associated computations. This trend has become unsustainable given the ever-growing use of deep learning, and could cause irreversible damage to the environment of our planet if it is not addressed soon. Objective:In this study, we aim to analyze not only the effects of different energy saving methods for neural networks but also the effects of the moment of intervention, and what makes certain moments optimal. Method:We developed a novel dataset by training convolutional neural networks in 12 different computer vision datasets and applying runtime decisions regarding layer freezing, model quantization and early stopping at different epochs in each run. We then fit an auto-regressive prediction model on the data collected capable to predict the accuracy and energy consumption achieved on future epochs for different methods. The predictions on accuracy and energy are used to estimate the optimal training path. Results:Following the predictions of the model can save 56.5% of energy consumed while also increasing validation accuracy by 2.38% by avoiding overfitting.The prediction model developed can predict the validation accuracy with a 8.4% of error, the energy consumed with a 14.3% of error and the trade-off between both with a 8.9% of error. Conclusions:This prediction model could potentially be used by the training algorithm to decide which methods apply to the model and at what moment in order to maximize the accuracy-energy trade-off.

1D -FNO - Residual convolutional neural network autoregressive prediction model for deep water wave train evolution based on high-order spectral (HOS) data

Accurate prediction of deep-water wave dynamics is essential for ocean engineering, meteorology, and maritime safety. Traditional physical models, though effective, face significant computational and time challenges with complex nonlinear dynamics. This study introduces an autoregressive prediction model using a one-dimensional adaptive Fourier Neural Operator-Residual Convolutional Neural Network (1D-FNO-ResNet) to enhance prediction accuracy. We developed a high-precision single-step prediction model capable of capturing wave characteristics such as height and fluid particle velocity. A fine-tuning training strategy based on multi-step prediction errors was proposed for autoregressive evolution. Performance comparisons between the ResNet and FNO-ResNet models were made under various fine-tuning steps. A correction model was also introduced to control error accumulation, improving autoregressive prediction steps. The model's generalization performance was validated with modulated and random wave trains under different conditions, optimizing predictive stability and accuracy. This approach demonstrates efficient 1D deep-water wave evolution prediction, offering a novel deep learning solution for wave dynamics prediction in ocean engineering, supporting wave energy conversion device design and optimization.

Direct prediction for oceanic mesoscale eddy geospatial distribution through prior statistical deep learning

Mesoscale Eddy (ME) is a widely recognized and significant oceanic phenomenon characterized by extensive energy exchange. Accurate predictions of the future geospatial distribution of ME are crucial for maritime activities. Deep learning (DL) methods for ME prediction have consistently outperformed classical mathematical models and numerical modeling approaches. However, current DL methods based on two-dimensional gridded data suffer from error accumulation issues due to their indirect prediction approach, which involves first predicting the future environmental field and then identifying the ME patterns within it. Additionally, these models ignore the potential influence of prior knowledge related to ME on prediction outcomes. To address these issues, this paper proposes a direct prediction DL network called ES-ConvGRU, which incorporates historical prior statistics. First, we conduct an analysis of ME data from the Northwest Pacific region, extracting prior statistical knowledge such as interannual variations and seasonal characteristics. Second, mining patterns for the spatiotemporal features of both ME sequence and sea surface environmental field sequence are established. Furthermore, a nonlinear approach for mapping prior statistical knowledge and a multi-step autoregressive prediction model are designed, facilitating the integration of prior knowledge with multidimensional spatiotemporal features for ME prediction. Finally, the performance and effectiveness of ES-ConvGRU are evaluated. It achieves an average pixel accuracy of 93.21%, a mean intersection over union of 81.48%, and a frequency-weighted intersection over union of 87.55% for the 7-day distribution prediction. The results show that ES-ConvGRU exhibits notable advantages compared to the existing indirect and direct prediction methods.

MARS: a motif-based autoregressive model for retrosynthesis prediction.

Retrosynthesis is a critical task in drug discovery, aimed at finding a viable pathway for synthesizing a given target molecule. Many existing approaches frame this task as a graph-generating problem. Specifically, these methods first identify the reaction center, and break a targeted molecule accordingly to generate the synthons. Reactants are generated by either adding atoms sequentially to synthon graphs or by directly adding appropriate leaving groups. However, both of these strategies have limitations. Adding atoms results in a long prediction sequence that increases the complexity of generation, while adding leaving groups only considers those in the training set, which leads to poor generalization. In this paper, we propose a novel end-to-end graph generation model for retrosynthesis prediction, which sequentially identifies the reaction center, generates the synthons, and adds motifs to the synthons to generate reactants. Given that chemically meaningful motifs fall between the size of atoms and leaving groups, our model achieves lower prediction complexity than adding atoms and demonstrates superior performance than adding leaving groups. We evaluate our proposed model on a benchmark dataset and show that it significantly outperforms previous state-of-the-art models. Furthermore, we conduct ablation studies to investigate the contribution of each component of our proposed model to the overall performance on benchmark datasets. Experiment results demonstrate the effectiveness of our model in predicting retrosynthesis pathways and suggest its potential as a valuable tool in drug discovery. All code and data are available at https://github.com/szu-ljh2020/MARS.

Nonlinear autoregressive models for high accuracy early prediction of Li-ion battery end-of-life

Predictions of the state-of-health (SOH) of Li-ion batteries is an important goal in the monitoring and management of electric vehicles. In recent years, a number of pure machine-learning methods have been proposed for such predictions. In this paper, we instead consider autoregression methods and embedding strategies, which are specifically tailored to time-series problems. For the first time, we comprehensively compare both linear and nonlinear approaches, including six deep learning architectures, autoregressive integrated moving average (ARIMA) models and seasonal ARIMA models. In particular, for the first time we introduce Gaussian process nonlinear autoregression (GPNAR) for SOH prediction and show that it is superior in terms of accuracy and computational costs to the other autoregressive approaches. On the basis of two different datasets, we also demonstrate that accurate early predictions of the end-of-life (based on 50% of the data) is achievable with GPNAR without the use of features, which keeps data acquisition and processing to a minimum. Finally, we show that GPNAR is capable of capturing seasonal trends such as regeneration without additional time-consuming data analyses. Comparisons to other state-of-the-art methods in the recent literature confirm the superior performance of GPNAR.

Rice crop growth analysis using Auto Regressive Models

Time series play a vital role in predicting and forecasting different types of agricultural applications with respect to different types of problems among successive units of observations. Time series forecasting techniques are applied in all areas of statistics, and one of the most important applications includes backscatter generating time-series data using advanced forecasting techniques. Agriculture is a major food sector in the world, and it is also a major income source for low-income people. In this paper, we present two aspects of the rice crop growing time series process. The first one is to identify different types of rice crop growing stages for backscatter datasets, and the second is to make a mathematical time series model for the generation of different data sets. The different operator techniques (DOT) method was introduced to identify different types of rice crop growing stages in a season. We proposed the DOT method for identification of different phenological stages for a short-term crop and adopted first and second-order auto-regressive models for prediction and forecasting of the generating backscatter time series observations. The measures of the quality fit are mean absolute percent error (MAPE), mean percent error (MPE), and mean absolute error (MAE).

Adaptive Gaussian Markov random field spatiotemporal models for infectious disease mapping and forecasting

Recent disease mapping literature presents adaptively parameterized spatiotemporal (ST) autoregressive (AR) or conditional autoregressive (CAR) models for Bayesian prediction of COVID-19 infection risks. These models were motivated to capture complex spatiotemporal dynamics and heterogeneities of infection risks. In the present paper, we synthesize, generalize, and unify the ST AR and CAR model constructions for models augmented by adaptive Gaussian Markov random fields, with an emphasis on disease forecasting. A general convolution construction is presented, with illustrative models motivated to (i) characterize local risk dependencies and influences over both spatial and temporal dimensions, (ii) model risk heterogeneities and discontinuities, and (iii) predict and forecast areal-level disease risks and occurrences. The broadened constructions allow rich options of intuitive parameterization for disease mapping and spatial regression. Illustrative parameterizations are presented for Bayesian hierarchical models of Poisson, zero-inflated Poisson, and Bernoulli data models, respectively. They are also discussed in the context of quantifying time-varying or time-invariant effects of (omitted) covariates, with application to prediction and forecasting areal-level COVID-19 infection occurrences and probabilities of zero-infection. The model constructions presented herein have much wider scope in offering a flexible framework for modelling complex spatiotemporal data and for estimation, learning, and forecasting purposes.

A physics-guided autoregressive model for saturation sequence prediction

A deep-learning-based surrogate model is developed and applied for predicting dynamic saturation sequences in oil-water two-phase reservoirs. Considering the limitations of most existing deep learning-based surrogate models in processing static and dynamic data simultaneously, and the lack of guidance from physical equations, a physics-guided autoregressive model is proposed to solve these problems. The surrogate model introduces the idea of explicit finite difference to make it work more in line with physical processes; the embedding of the mass balance equation makes its prediction accuracy significantly improved. To evaluate the performance of the proposed method, three representative models are comprehensively compared, namely the recurrent R-U-Net, the autoregressive model without the physical meaning embedded and the physics-guided autoregressive model. And experiments show that the physics-guided autoregressive model is capable of predicting accurate dynamic saturation maps and achieves very competitive results. Compared to the numerical simulation, the trained surrogate model is capable of predicting the saturation sequence efficiently, quickly, and accurately. We believe it has the potential to replace the forward process of numerical simulation when predicting saturation.

Comparison metrics for multi-step prediction of rare events in vital sign signals

Prediction of changes in biomedical signals, such as vital signs, is useful for many clinical applications. Several signal prediction (forecasting) tools were developed, but their evaluation and applicability to a specific clinical use is context dependent. In this work, we propose a novel method to tackle the problem of evaluation and comparison of vital sign predictors for intervention based clinical studies. The proposed prediction quality measures are particularly well-suited for forecasting rare events scenarios. Specifically, using the novel metrics, we measure the prediction statistics and compare nine deep learning and autoregressive forecasting models for multi-step prediction of rare bradycardia events in preterm infants, however the new concepts allow applications to other biomedical signals. We validated the novel metrics with experimental results on testing sets with several days of vital sign recordings. Our results show that simple statistical predictors could outperform state-of-the-art deep learning architectures for low-dimensional signals.

Predicting Drying Curves in Algal Biorefineries using Gaussian Process Autoregressive Models

In algal biofuel production, drying of the microalgal biomass is considered the most energy-intensive process. To save on operating costs, optimal control of the drying process should be guided by accurate mathematical models of the biomass parameters, particularly its moisture content. In this paper, we propose the use of Gaussian process autoregressive models (GPAR) for long-range moisture content prediction in the vacuum drying of algal biofuels. Our experiments involve the drying of Chlorococcum infusionum, wherein the measured variables are temperature, pressure, and moisture ratio. By computing the root mean squared error (RMSE) on test data, we demonstrate the superiority of GPAR to other models, namely Neural Networks, Support Vector Machines, Random Forest, Gradient Boosting, and Autoregressive Models for the same task. Through automatic relevance determination kernels, GPAR also found that the most significant predictors are the pressure readings. In the future, GPAR can potentially be used for the predictive control of the drying process, leading to more efficient biorefinery operations.

A hybrid deep learning approach by integrating extreme gradient boosting‐long short‐term memory with generalized autoregressive conditional heteroscedasticity family models for natural gas load volatility prediction

AbstractNatural gas load forecasting provides decision‐making support for natural gas dispatch and management, pipeline network construction, pricing, and sustainable energy development. To explain the uncertainty and volatility in natural gas load forecasting, this study predicts the natural gas load volatility. As the natural gas load volatility has the time‐series features, along with long‐term memory, volatility aggregation, asymmetry, and nonnormality, this study proposes a natural gas load volatility prediction model by combining generalized autoregressive conditional heteroscedasticity (GARCH) family models, XGBoost algorithm, and long short‐term memory (LSTM) network. The model first takes the GARCH family models parameters of sliding estimation and meteorological factors as the influencing factors of volatility, and then it screens these influencing factors through the extreme gradient boosting (XGBoost) algorithm. Finally, the selected important features are input into the LSTM network to predict the volatility, and the 90% confidence interval of the volatility is calculated. Compared with a variety of single and combined models, the model proposed in this study has an average reduction of 45.404% in the evaluation index of mean squared error. The experimental results show that the model proposed in this study has a good performance and accuracy in predicting the volatility of natural gas load.

Optimized Intelligent Auto-Regressive Neural Network Model (ARNN) for Prediction of Non-Linear Exogenous Signals

The paper presents a prediction of non-linear exogenous signal by optimized intelligent auto-regressive neural network model (ARNN). A signal comprises of two sets of data called deterministic and error. The former type of data represents the degradation index of a signal, while the error is the uncertainties associated with it. To understand and predict signals, an intelligent approach is taken using ARNN model. The deterministic component is predicted by developing a neural network based non-linear autoregressive model and the error component by using a linear stochastic model. The final forecast is formed by combining the results from each of the models and evaluated using the mean square error results. Validation of the prediction is obtained through a comparison of the results with existing models. It shows that ARNN reduced the MSE error 14-36\(\%\), RMSS error 19-46\(\%\) and NMSE 18-42\(\%\) as compared to existing models. Moreover, ARNN model can be used for low as well as for high volatility data elements. The results show that the proposed model provides improved predictions and minimizes high dependence on design parameters with low computational cost.

Decomposition-based auto-regressive model for agricultural price prediction

Decomposition-based auto-regressive model for agricultural price prediction

Model framework to quantify the effectiveness of garbage classification in reducing dioxin emissions

Although waste incineration is a promising disposal method, it produces unwanted combustion by-products, such as toxic dioxins, that can be unintentionally emitted. Kitchen scraps can result in incomplete combustion of waste, which accelerates the formation of dioxins, especially for the small-sized incinerators without identical operating temperature. Consequently, garbage classification before waste incineration is critical for dioxin control in the small-sized waste incineration industries. To date, the influence of garbage classification on dioxin emissions has not been quantified. In this study, a model framework integrating the grey prediction model and autoregressive prediction model was established and used to predict future dioxin emissions from small-sized waste incineration. If garbage classification is ideally strictly implemented, annual dioxin emissions could be reduced by up to 1697 g TEQ over the next 10 years. Garbage classification reduced emissions by about 30.7% compared with incineration of mixed municipal solid waste without classification (5534 g TEQ over the next 10 years). The established model framework can effectively assess the influence of garbage classification on dioxin emissions from waste incineration, which could facilitate the widespread adoption of garbage classification.

An integrated autoregressive model for predicting water quality dynamics and its application in Yongding River

It is important to predict evolution processes of water quality in order to effectively protect the aquatic eco-system. We proposed an integrated autoregressive (AR) prediction model based on Markov-switching (MS) theory and the improved crow search algorithm (ICSA), and the integrated model of ICSA-MSAR was used to predict temporal evolution of water quality changes. The results showed that the MS and ICSA provided an optimized method to determine the model parameters, and the integrated model with (s, p) values of (3, 5) had a better performance. Compared with the traditional AR model, the integrated ICSA-MSAR model significantly raised Akaike information criterion by 32.0%, Bayesian information criterion by 14.2%, Log-likelihood value by 40.5%, C index (C=AIC×BIC×LL) by 65.6%, and reduced the prediction relative error by 73.3%. The better performance mainly attributed to its offset capacity and convergence capacity. The integrated model can well predict the changes of CODMn in Yongding River located of North China during 2008–2018, and it is worth extending this model to predict the changes of other ecological indicators in water bodies.

A hybrid autoregressive fractionally integrated moving average and nonlinear autoregressive neural network model for short-term traffic flow prediction

Intelligent traffic control and guidance system is an effective way to solve urban traffic congestion, improve road capacity and guarantee drivers' travel safety, while short-term traffic flow prediction is the core of intelligent traffic control and guidance system. To investigate the long-term memory and the dynamic feature of short-time traffic flow time series, a hybrid model was proposed by integrating autoregressive fractionally integrated moving average (ARFIMA) model and nonlinear autoregressive (NAR) neural network model to predict short-time traffic flow, in which ARFIMA model can address the long-term memory of linear component and NAR neural network can accommodate the dynamic feature of nonlinear residual component. First, the ARFIMA model was employed to predict the linear component of traffic flow, and the results were compared with those of autoregressive integrated moving average (ARIMA) model. Next, the NAR neural network model was adopted to forecast the nonlinear residual components, and the weighted results were considered as the predicted flow of the hybrid model. The proposed hybrid model was validated by using the cross-sectional traffic flow data in California freeways obtained from the open-access PeMS database. The results showed that the ARFIMA model considering the long-term memory can effectively predict the short-term traffic flow, and the prediction accuracy of the hybrid model is better than that of the singular models. The findings provide an alternative for the short-term traffic flow prediction with lower error and higher accuracy.

A time-domain approach to the total ozone time series and a test of its predictability within a univariate framework

ABSTRACT The present study reports a univariate predictive model for Total Ozone (TO) concentration derived from Ozone Monitoring Instrument (OMI)/Aura observations. Using the Markovian approach through proper discretization method it has been observed that the second-order Markov Chain can represent the time series of TO Concentration. Considering daily data spanning from 2015 to 2019, consisting of 1593 daily TO concentration data. Identifying second order as the most suitable Markov Chain we have considered a second-order autoregressive model for univariate prediction of the time series. Finally, the prediction of TO concentration available from satellite remote sensing has been assessed statistically for its prediction accuracy. The prediction accuracy has been assessed through Willmott’s Indices and Pearson Correlation Coefficient.

Improving Tourist Arrival Prediction: A Big Data and Artificial Neural Network Approach

Because of high fluctuations of tourism demand, accurate predictions of tourist arrivals are of high importance for tourism organizations. The study at hand presents an approach to enhance autoregressive prediction models by including travelers’ web search traffic as external input attribute for tourist arrival prediction. The study proposes a novel method to identify relevant search terms and to aggregate them into a compound web-search index, used as additional input of an autoregressive prediction approach. As methods to predict tourism arrivals, the study compares autoregressive integrated moving average (ARIMA) models with the machine learning–based technique artificial neural network (ANN). Study results show that (1) Google Trends data, mirroring traveler’s online search behavior (i.e., big data information source), significantly increase the performance of tourist arrival prediction compared to autoregressive approaches using past arrivals alone, and (2) the machine learning technique ANN has the capacity to outperform ARIMA models.

Model Predictive Control of a Shipborne Hydraulic Parallel Stabilized Platform Based on Ship Motion Prediction

Shipborne stabilized platform is an important equipment to ensure the stability of shipborne equipment relative to inertial coordinate system. This paper presents a model predictive control strategy based on ship motion prediction (MPMPC) for ship stabilization platform. Firstly, the ship motion is simulated, and the autoregressive prediction model (AR model) is used to predict the ship motion. Then the kinematics analysis of the Shipborne stabilized platform is carried out and the mathematical model of the hydraulic drive unit (HDU) of the stable platform is established. Then the predicted ship motion is combined with model predictive control (MPC). The predicted trajectory of HDU can be obtained by the kinematics calculation of predicted ship motion. One part of the predicted trajectory is used to compensate the time delay of HDU, and the other part is used as the reference trajectory of the rolling optimization of MPC, instead of the reference trajectory using the measured ship motion at the current moment in traditional model predictive control. Compared with the reference trajectory using the measured ship motion at the current moment, the predicted trajectory of AR model can reflect the future state of the system better, and a better control sequence will be obtained by minimizing the objective function. Finally, the simulation and experiment show that the MPMPC has higher tracking accuracy than traditional MPC.

Повышение точности предсказания цены электроэнергии за счет применения комбинированных моделей прогноза

The solution of problems concerned with predicting a free market price for electricity through constructing different prediction models is considered. In so doing, a shift is made from an analysis of conventional regression and auto-regression models of the moving average to the proposed combined multifactor models, which also include the time trend and dummy variables. This shift is partly justified by the specific behavior of the electricity price in the free market, which is caused by a strictly cyclic change of its value, e.g., proceeding from such attributes as the heating season, day of week, etc. The techniques of constructing combined prediction models has been developed to the level of elaborating effective computational procedures based on the Statistica and OsiSoft PI-System software packages. The application of the autoregressive and combined regression prediction models to the Russian market has demonstrated their fairly good effectiveness with an acceptable level of accuracy. A comparison of the achieved levels of accuracy provided by the competing models has not shown any advantages of the shift to the use of combined regression multifactor models in terms of achieving better prediction accuracy; however, their application for analyzing the influence of different factors on the predicted variable may become a fundamental advantage in selecting the type of prediction model. Despite their being limited to an analysis of the Belgorod region market, the obtained results demonstrate the achieved prediction accuracy that is as least as good, and in the main is even better than the majority of the data presented in the review of the results for European electricity markets. The article substantiates the advisability of studying the combined regression models as a tool for analyzing the influence of individual factors as components influencing the electricity price formation for the predicted period, given that the accuracy level of the combined regression models corresponds to the currently achieved electricity price prediction accuracy levels.