Time Series Cross-validation Research Articles

What factors influence Bitcoin’s daily price direction from the perspective of machine learning classifiers?

The paper examines the factors that influence Bitcoin price direction from the perspective of machine learning (ML) models. The observed factors cover Bitcoin market data, technical indicators, blockchain variables, sentiment analysis, and other macro-financial variables. Logistic Regression (LR), Random Forest (RF), and Support Vector Machine (SVM) classifiers are employed. Three train-test ratios are considered. Grid search and blocking time series cross-validation are used to adjust the hyperparameters of the proposed ML algorithms resulting in the three most accurate models for each train-test ratio. Variables that affect the next-day price direction are ranked using LR and RF best models. For each method and train-test ratio, the smallest subsets of independent variables with the highest test set accuracy were chosen to reduce dimensionality. Models show that technical indicators influence daily Bitcoin price direction the most, followed by blockchain and Bitcoin market variables. Contrarily, models disagree on the importance of Tweets and macro-financial variables. Finally, SVM performed better on the test set when the LR optimal sets of independent variables were considered, indicating that the analysis of individual factors' influence on the Bitcoin price is not important only for corresponding model. Combining only influential independent variables and 90:10 train-test ratio yielded the greatest accuracy of 58.18 % achieved by RF model.

Long-Term Projections of Cancer Incidence and Mortality in Japan and Decomposition Analysis of Changes in Cancer Burden, 2020-2054: An Empirical Validation Approach.

The aim of this study was to project new cancer cases/deaths forward to 2054, and decompose changes in cancer cases/deaths to assess the impact of demographic transitions on cancer burden. We collected data on cancer cases/deaths up to 2019, empirically validated the projection performance of multiple statistical models, and selected optimal models by applying time series cross-validation. We showed an increasing number of new cancer cases but decreasing number of cancer deaths in both genders, with a large burden attributed to population aging. We observed the increasing incidence rates in most cancer sites but reducing rates in some infection-associated cancers, including stomach and liver cancers. Colorectal and lung cancers were projected to remain as leading cancer burdens of both incidence and mortality in Japan over 2020-2054, while prostate and female breast cancers would be the leading incidence burdens among men and women, respectively. Findings from decomposition analysis require more supportive interventions for reducing mortality and improving the quality of life of Japanese elders. We emphasize the important role of governments and policymakers in reforming policies for controlling cancer risk factors, including oncogenic infections. The rapid increase and continued presence of those cancer burdens associated with modifiable risk factors warrant greater efforts in cancer control programs, specifically in enhancing cancer screening and controlling cancer risk factors in Japan.

Forecasting the Direction of Daily Changes in the India VIX Index Using Machine Learning

Movements in the India VIX are an important gauge of how the market’s risk perception shifts from day to day. This research attempts to forecast movements one day ahead of the India VIX using logistic regression and 11 ensemble learning classifiers. The period of study is from April 2009 to March 2021. To achieve the stated task, classifiers were trained and validated with 90% of the given sample, considering two-fold time-series cross-validation for hyper-tuning. Optimised models were then predicted on an unseen test dataset, representing 10% of the given sample. The results showed that optimal models performed well, and their accuracy scores were similar, with minor variations ranging from 63.33% to 67.67%. The stacking classifier achieved the highest accuracy. Furthermore, CatBoost, Light Gradient Boosted Machine (LightGBM), Extreme Gradient Boosting (XGBoost), voting, stacking, bagging and Random Forest classifiers are the best models with statistically similar performances. Among them, CatBoost, LightGBM, XGBoost and Random Forest classifiers can be recommended for forecasting day-to-day movements of the India VIX because of their inherently optimised structure. This finding is very useful for anticipating risk in the Indian stock market.

Deep Learning Methods for Predicting Tap-Water Quality Time Series in South Korea

South Korea currently lacks a real-time monitoring and anomaly detection system for detecting continuous tap water quality changes from the water source to faucet and pre-diagnosing hazards that threaten tap water safety. In this study, we constructed an accurate water quality prediction model that could comprehensively cover all water treatment facilities supplying tap water nationwide and verified the model using an integrated approach. To address the uncertainty of continuously changing water quality, we collected five years (2017–2021) of hourly water quality data from 33 large water purification plants and applied various deep learning techniques to construct an optimal prediction model. We repeated water quality prediction and evaluation over the following 24 h through a time series cross-validation of an untrained dataset of the previous five months. The optimized deep learning model achieved average and maximum prediction accuracy of 98.78 and 99.98%, respectively, and showed excellent performance in terms of the root mean squared error (0.0006), mean absolute error (0.0003), and Nash–Sutcliffe efficiency (0.9894). Thus, deep learning technology greatly improved the accuracy and efficiency of water quality prediction. The proposed model could provide prompt and accurate water quality information for large-scale water supply facilities nationwide and improve public health through the early diagnosis of water quality anomalies.

Effective machine learning model combination based on selective ensemble strategy for time series forecasting

The success of ensemble forecasting heavily depends on the selection and combination of component models as proven by numerous studies that show the superior performance of ensemble forecasting in modeling volatile time series. This work develops a selective machine learning ensemble (SMLE) model that evaluates the validation accuracy of the model ensemble by applying a time series cross-validation method, and establish an efficient model combination process by using a novel soft selection algorithm for selecting and weighing several machine learning models, such as support vector regression, feedforward neural network, random forest, and gradient boosting decision tree. The well-known NN3 time series forecasting experiment and a real-world crude oil price forecasting application are used to verify the performance of SMLE. Numerical results and in-depth analysis indicate that the proposed model can provide more accurate and reliable predictions compared with those obtained via individual forecasts, advanced forecasting techniques, and ensemble strategies.

Estimation of dominant power oscillation modes based on ConvLSTM approach using synchrophasor data and cross-validation technique

This paper proposes a deep neural network approach considering performance-based cross-validation and confidence interval analysis to estimate a power system’s dominant power oscillation modes. Due to increased electricity demands, power utilities implement various generation sources in their power systems. Accordingly, a modern power system is increasingly complex as a multi-area and multi-machine power system. The electromechanical oscillation modes arise inevitably. Moreover, a major-unexpected event could excite weakly damped power oscillation modes and cause power system instability. The estimation of dominant power oscillation modes is significant for power system monitoring and control. A fast computing time of such modes estimation is essential for further actions. This paper applies the convolutional long short-term memory 2-dimension (ConvLSTM2D) approach to estimate dominant oscillation modes based on synchrophasor data. The proposed ConvLSTM2D approach provides precise estimation with a great opportunity to avoid a forced power system outage. The simulation results of the ConvLSTM2D approach show better accuracy of the dominant power oscillation modes estimated in comparison with the state-of-the-art algorithms (SOTA), i.e., long short-term memory (LSTM), gated recurrent unit (GRU), and hybrid convolutional neural networks-long short-term memory (CNN-LSTM) algorithms. The proposed approach is a systematic approach that can be adaptively improved over time. In addition, the proposed approach can be further applied to wide-area monitoring considering the stability margin of a transmission system. • A ConvLSTM approach is proposed to estimate multiple dominant power oscillation modes. • The proposed approach using time-series cross-validation provides a promising performance. • The performance of proposed approach is robust proved by confidence interval analysis. • The proposed approach can reduce model parameters and training time.

Ensemble Machine Learning of Random Forest, AdaBoost and XGBoost for Vertical Total Electron Content Forecasting

Space weather describes varying conditions between the Sun and Earth that can degrade Global Navigation Satellite Systems (GNSS) operations. Thus, these effects should be precisely and timely corrected for accurate and reliable GNSS applications. That can be modeled with the Vertical Total Electron Content (VTEC) in the Earth’s ionosphere. This study investigates different learning algorithms to approximate nonlinear space weather processes and forecast VTEC for 1 h and 24 h in the future for low-, mid- and high-latitude ionospheric grid points along the same longitude. VTEC models are developed using learning algorithms of Decision Tree and ensemble learning of Random Forest, Adaptive Boosting (AdaBoost), and eXtreme Gradient Boosting (XGBoost). Furthermore, ensemble models are combined into a single meta-model Voting Regressor. Models were trained, optimized, and validated with the time series cross-validation technique. Moreover, the relative importance of input variables to the VTEC forecast is estimated. The results show that the developed models perform well in both quiet and storm conditions, where multi-tree ensemble learning outperforms the single Decision Tree. In particular, the meta-estimator Voting Regressor provides mostly the lowest RMSE and the highest correlation coefficients as it averages predictions from different well-performing models. Furthermore, expanding the input dataset with time derivatives, moving averages, and daily differences, as well as modifying data, such as differencing, enhances the learning of space weather features, especially over a longer forecast horizon.

A machine learning framework to predict the next month's daily milk yield, milk composition and milking frequency for cows in a robotic dairy farm

Robotic milking systems (RMS) are increasingly utilised by modern livestock farmers because they can reduce labour costs, and they have the potential to collect data that will improve animal welfare and animal productivity through better monitoring. Sensors and devices installed in RMS enable farmers to routinely collect data on environment conditions, individual animal's behaviours, health, productivity, and milk quality. This dataset can be used to train artificial intelligence algorithms to predict trends in these variables. This study developed a machine learning framework using 5 years' behaviour, heath and productivity data from 80 cows in a robotic dairy farm. Here we demonstrate the development of a framework to automatically train models with up-to-date farm data and predict daily milk yield, composition (fat and protein content) and frequency of individual cow milking during the subsequent 28 days. A time series cross-validation was applied to simulate the application of this framework under commercial conditions and to evaluate the performance. A high accuracy of prediction (R2 > 0.90 and overall accuracy > 80%) was achieved with the models created by this framework. The practical potential of using such frameworks to enhance the management efficiency and animal welfare in robotic dairy farms is discussed.

Prediction of clinical trial enrollment rates.

Clinical trials represent a critical milestone of translational and clinical sciences. However, poor recruitment to clinical trials has been a long standing problem affecting institutions all over the world. One way to reduce the cost incurred by insufficient enrollment is to minimize initiating trials that are most likely to fall short of their enrollment goal. Hence, the ability to predict which proposed trials will meet enrollment goals prior to the start of the trial is highly beneficial. In the current study, we leveraged a data set extracted from ClinicalTrials.gov that consists of 46,724 U.S. based clinical trials from 1990 to 2020. We constructed 4,636 candidate predictors based on data collected by ClinicalTrials.gov and external sources for enrollment rate prediction using various state-of-the-art machine learning methods. Taking advantage of a nested time series cross-validation design, our models resulted in good predictive performance that is generalizable to future data and stable over time. Moreover, information content analysis revealed the study design related features to be the most informative feature type regarding enrollment. Compared to the performance of models built with all features, the performance of models built with study design related features is only marginally worse (AUC = 0.78 ± 0.03 vs. AUC = 0.76 ± 0.02). The results presented can form the basis for data-driven decision support systems to assess whether proposed clinical trials would likely meet their enrollment goal.

Interpretable Short-Term Electrical Load Forecasting Scheme Using Cubist

Daily peak load forecasting (DPLF) and total daily load forecasting (TDLF) are essential for optimal power system operation from one day to one week later. This study develops a Cubist-based incremental learning model to perform accurate and interpretable DPLF and TDLF. To this end, we employ time-series cross-validation to effectively reflect recent electrical load trends and patterns when constructing the model. We also analyze variable importance to identify the most crucial factors in the Cubist model. In the experiments, we used two publicly available building datasets and three educational building cluster datasets. The results showed that the proposed model yielded averages of 7.77 and 10.06 in mean absolute percentage error and coefficient of variation of the root mean square error, respectively. We also confirmed that temperature and holiday information are significant external factors, and electrical loads one day and one week ago are significant internal factors.

Forecasting Reservoir Water Levels Using Deep Neural Networks: A Case Study of Angat Dam in the Philippines

Forecasting reservoir water levels is essential in water supply management, impacting both operations and intervention strategies. This paper examines the short-term and long-term forecasting performance of several statistical and machine learning-based methods for predicting the water levels of the Angat Dam in the Philippines. A total of six forecasting methods are compared: naïve/persistence; seasonal mean; autoregressive integrated moving average (ARIMA); gradient boosting machines (GBM); and two deep neural networks (DNN) using a long short-term memory-based (LSTM) encoder-decoder architecture: a univariate model (DNN-U) and a multivariate model (DNN-M). Daily historical water levels from 2001 to 2021 are used in predicting future water levels. In addition, we include meteorological data (rainfall and the Oceanic Niño Index) and irrigation data as exogenous variables. To evaluate the forecast accuracy of our methods, we use a time series cross-validation approach to establish a more robust estimate of the error statistics. Our results show that our DNN-U model has the best accuracy in the 1-day-ahead scenario with a mean absolute error (MAE) and root mean square error (RMSE) of 0.2 m. In the 30-day-, 90-day-, and 180-day-ahead scenarios, the DNN-M shows the best performance with MAE (RMSE) scores of 2.9 (3.3), 5.1 (6.0), and 6.7 (8.1) meters, respectively. Additionally, we demonstrate that further improvements in performance are possible by scanning over all possible combinations of the exogenous variables and only using a subset of them as features. In summary, we provide a comprehensive framework for evaluating water level forecasting by defining a baseline accuracy, analyzing performance across multiple prediction horizons, using time series cross-validation to assess accuracy and uncertainty, and examining the effects of exogenous variables on forecasting performance. In the process, our work addresses several notable gaps in the methodologies of previous works.

Forests for Fama

In this article, the author addresses Eugene Fama’s skepticism regarding the predictability of stock market bubbles. To do so, he applies two ensemble learning methods, the random cut forest and random forest algorithms, to build a model that predicts large near-term drawdowns based on patterns in stock price behavior. The model includes three predictive variables. The first factor is an anomaly score produced by random cut forest, an algorithm specifically designed to detect outliers in streaming data. The second and third factors are the standard deviation of price returns and the return convexity over specified time horizons, with return convexity defined as the difference between one-year price returns and six-month price returns. The author’s predictions are based on random forest regressions. He applies the model to a large universe of equity sectors and factors. Blocked time-series cross-validation is used to evaluate the predictive efficacy of the model. The author shows that across the sectors and factors considered, the model presented produces predictive scores that are strongly positive. Although bubble prediction is surely a multidimensional endeavor that requires input from a variety of tools and sources, the author demonstrates that a framework built upon ensemble methods can be informationally additive to the detection of bubblelike behavior across a wide array of stocks.

Beyond the beaten paths of forecasting call center arrivals: on the use of dynamic harmonic regression with predictor variables

Modern call centers require precise forecasts of call and e-mail arrivals to optimize staffing decisions and to ensure high customer satisfaction through short waiting times and the availability of qualified agents. In the dynamic environment of multi-channel customer contact, organizational decision-makers often rely on robust but simplistic forecasting methods. Although forecasting literature indicates that incorporating additional information into time series predictions adds value by improving model performance, extant research in the call center domain barely considers the potential of sophisticated multivariate models. Hence, with an extended dynamic harmonic regression (DHR) approach, this study proposes a new reliable method for call center arrivals’ forecasting that is able to capture the dynamics of a time series and to include contextual information in form of predictor variables. The study evaluates the predictive potential of the approach on the call and e-mail arrival series of a leading German online retailer comprising 174 weeks of data. The analysis involves time series cross-validation with an expanding rolling window over 52 weeks and comprises established time series as well as machine learning models as benchmarks. The multivariate DHR model outperforms the compared models with regard to forecast accuracy for a broad spectrum of lead times. This study further gives contextual insights into the selection and optimal implementation of marketing-relevant predictor variables such as catalog releases, mail as well as postal reminders, or billing cycles.

On Comparing Cross-Validated Forecasting Models with a Novel Fuzzy-TOPSIS Metric: A COVID-19 Case Study

Time series cross-validation is a technique to select forecasting models. Despite the sophistication of cross-validation over single test/training splits, traditional and independent metrics, such as Mean Absolute Error (MAE) and Root Mean Square Error (RMSE), are commonly used to assess the model’s accuracy. However, what if decision-makers have different models fitting expectations to each moment of a time series? What if the precision of the forecasted values is also important? This is the case of predicting COVID-19 in Amapá, a Brazilian state in the Amazon rainforest. Due to the lack of hospital capacities, a model that promptly and precisely responds to notable ups and downs in the number of cases may be more desired than average models that only have good performances in more frequent and calm circumstances. In line with this, this paper proposes a hybridization of the Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) and fuzzy sets to create a similarity metric, the closeness coefficient (CC), that enables relative comparisons of forecasting models under heterogeneous fitting expectations and also considers volatility in the predictions. We present a case study using three parametric and three machine learning models commonly used to forecast COVID-19 numbers. The results indicate that the introduced fuzzy similarity metric is a more informative performance assessment metric, especially when using time series cross-validation.

A Two-Stage Multistep-Ahead Electricity Load ForecastingScheme Based on LightGBM and Attention-BiLSTM.

An efficient energy operation strategy for the smart grid requires accurate day-ahead electricity load forecasts with high time resolutions, such as 15 or 30 min. Most high-time resolution electricity load prediction techniques deal with a single output prediction, so their ability to cope with sudden load changes is limited. Multistep-ahead forecasting addresses this problem, but conventional multistep-ahead prediction models suffer from deterioration in prediction performance as the prediction range is expanded. In this paper, we propose a novel two-stage multistep-ahead forecasting model that combines a single-output forecasting model and a multistep-ahead forecasting model to solve the aforementioned problem. In the first stage, we perform a single-output prediction based on recent electricity load data using a light gradient boosting machine with time-series cross-validation, and feed it to the second stage. In the second stage, we construct a multistep-ahead forecasting model that applies an attention mechanism to sequence-to-sequence bidirectional long short-term memory (S2S ATT-BiLSTM). Compared to the single S2S ATT-BiLSTM model, our proposed model achieved improvements of 3.23% and 4.92% in mean absolute percentage error and normalized root mean square error, respectively.

A Data-Driven Framework for Automated Detection of Aircraft-Generated Signals in Seismic Array Data Using Machine Learning

Abstract Ground motions associated with aircraft overflights can cover a significant portion of the seismic data collected by shallowly emplaced seismometers, such as new nodal and Distributed Acoustic Sensing systems. This article describes the first published framework for automated detection of aircraft on single channel and multichannel seismic data. The seismic data are converted to spectrograms in a sliding time window and classified as aircraft or nonaircraft in each window using a deep convolutional neural network trained with analyst-labeled data. A majority voting scheme is used to convert the output from the sequence of sliding time windows onto a decision time sequence for each channel and to combine the binary classifications on the decision time sequences across multiple channels. Precision, recall, and F-score are used to quantify the detection performance of the algorithm on nodal data using fourfold time-series cross validation. By applying our framework to data from the Sage Brush Flats nodal array in Southern California, we provide a benchmark performance and demonstrate the advantage of using an array of sensors.

Forecasting COVID-19 Hospital Census: A Multivariate Time-Series Model Based on Local Infection Incidence.

BackgroundCOVID-19 has been one of the most serious global health crises in world history. During the pandemic, health care systems require accurate forecasts for key resources to guide preparation for patient surges. Forecasting the COVID-19 hospital census is among the most important planning decisions to ensure adequate staffing, number of beds, intensive care units, and vital equipment.ObjectiveThe goal of this study was to explore the potential utility of local COVID-19 infection incidence data in developing a forecasting model for the COVID-19 hospital census.MethodsThe study data comprised aggregated daily COVID-19 hospital census data across 11 Atrium Health hospitals plus a virtual hospital in the greater Charlotte metropolitan area of North Carolina, as well as the total daily infection incidence across the same region during the May 15 to December 5, 2020, period. Cross-correlations between hospital census and local infection incidence lagging up to 21 days were computed. A multivariate time-series framework, called the vector error correction model (VECM), was used to simultaneously incorporate both time series and account for their possible long-run relationship. Hypothesis tests and model diagnostics were performed to test for the long-run relationship and examine model goodness of fit. The 7-days-ahead forecast performance was measured by mean absolute percentage error (MAPE), with time-series cross-validation. The forecast performance was also compared with an autoregressive integrated moving average (ARIMA) model in the same cross-validation time frame. Based on different scenarios of the pandemic, the fitted model was leveraged to produce 60-days-ahead forecasts.ResultsThe cross-correlations were uniformly high, falling between 0.7 and 0.8. There was sufficient evidence that the two time series have a stable long-run relationship at the .01 significance level. The model had very good fit to the data. The out-of-sample MAPE had a median of 5.9% and a 95th percentile of 13.4%. In comparison, the MAPE of the ARIMA had a median of 6.6% and a 95th percentile of 14.3%. Scenario-based 60-days-ahead forecasts exhibited concave trajectories with peaks lagging 2 to 3 weeks later than the peak infection incidence. In the worst-case scenario, the COVID-19 hospital census can reach a peak over 3 times greater than the peak observed during the second wave.ConclusionsWhen used in the VECM framework, the local COVID-19 infection incidence can be an effective leading indicator to predict the COVID-19 hospital census. The VECM model had a very good 7-days-ahead forecast performance and outperformed the traditional ARIMA model. Leveraging the relationship between the two time series, the model can produce realistic 60-days-ahead scenario-based projections, which can inform health care systems about the peak timing and volume of the hospital census for long-term planning purposes.

Analysis of Lumber Prices Time Series Using Long Short-Term Memory Artificial Neural Networks

This manuscript confirms the feasibility of using a long short-term memory (LSTM) recurrent neural network (RNN) to forecast lumber stock prices during the great and Coronavirus disease 2019 (COVID-19) pandemic recessions in the USA. The database was composed of 5012 data entries divided into recession periods. We applied a timeseries cross-validation that divided the dataset into an 80:20 training/validation ratio. The network contained five LSTM layers with 50 units each followed by a dense output layer. We evaluated the performance of the network via mean squared error (MSE), root mean squared error (RMSE), and mean absolute error (MAE) for 30, 60, and 120 timesteps and the recession periods. The metrics results indicated that the network was able to capture the trend for both recession periods with a remarkably low degree of error. Timeseries forecasting may help the forest and forest product industries to manage their inventory, transportation costs, and response readiness to critical economic events.

Operational solar flare prediction model using Deep Flare Net

We developed an operational solar flare prediction model using deep neural networks, named Deep Flare Net (DeFN). DeFN can issue probabilistic forecasts of solar flares in two categories, such as ≥ M-class and < M-class events or ≥ C-class and < C-class events, occurring in the next 24 h after observations and the maximum class of flares occurring in the next 24 h. DeFN is set to run every 6 h and has been operated since January 2019. The input database of solar observation images taken by the Solar Dynamic Observatory (SDO) is downloaded from the data archive operated by the Joint Science Operations Center (JSOC) of Stanford University. Active regions are automatically detected from magnetograms, and 79 features are extracted from each region nearly in real time using multiwavelength observation data. Flare labels are attached to the feature database, and then, the database is standardized and input into DeFN for prediction. DeFN was pretrained using the datasets obtained from 2010 to 2015. The model was evaluated with the skill score of the true skill statistics (TSS) and achieved predictions with TSS = 0.80 for ≥ M-class flares and TSS = 0.63 for ≥ C-class flares. For comparison, we evaluated the operationally forecast results from January 2019 to June 2020. We found that operational DeFN forecasts achieved TSS = 0.70 (0.84) for ≥ C-class flares with the probability threshold of 50 (40)%, although there were very few M-class flares during this period and we should continue monitoring the results for a longer time. Here, we adopted a chronological split to divide the database into two for training and testing. The chronological split appears suitable for evaluating operational models. Furthermore, we proposed the use of time-series cross-validation. The procedure achieved TSS = 0.70 for ≥ M-class flares and 0.59 for ≥ C-class flares using the datasets obtained from 2010 to 2017. Finally, we discuss the standard evaluation methods for operational forecasting models, such as the preparation of observation, training, and testing datasets, and the selection of verification metrics.

Time Series Cross Validation: Theoretical Properties and Empirical Performance

Time Series Cross Validation: Theoretical Properties and Empirical Performance