Forecast Error Measures Research Articles

From Hindsight to Foresight: Understanding and Overcoming Bias in Financial Forecasts

Abstract This study conducts a comprehensive benchmarking analysis on analyst forecasts of revenue and EBITDA for exchange-traded public companies in the U.S. spanning the period from 2009 to 2022. Our findings reveal a consistent and statistically significant upward bias present in both revenue and EBITDA forecasts, on average, as evidenced by various measures of forecast error. This bias is explored at both market sector and market-wide levels of aggregation to check for macro-consistency. Recognizing the significance of these biases, we propose bias-adjustment frameworks aimed at enhancing the accuracy of future analyst forecasts pertaining to revenue and EBITDA. The implications of our study underscore the importance of mitigating such biases for improved forecasting precision in financial markets.

Predictions of Malaysia Age-Specific Fertility Rates using the Lee-Carter and the Functional Data Approaches

Global fertility has been experiencing a significant decline, reaching towards the replacement ratio. This trend, coupled with increasing life expectancies, has led to the emergence of an ageing population. In this study, we delve into an analysis of fertility patterns among Malaysian women, considering both their childbearing age and ethnic groups. A comprehensive 63-year fertility dataset, from 1958 to 2020, were obtained from the Department of Statistics Malaysia. These data were fitted into the Lee-Carter model and its modified version, which is the functional data model. The models were evaluated using the out-sample forecast error measures. Results indicate that the third-order functional data model able to capture most of variation present in the actual data, consequently outperforming the Lee-Carter model in forecasting fertility rates among Chinese and Indian populations. The 20-year forecasts reveal a noteworthy shift in maternal ages of the highest births to older ages suggesting a trend towards delayed pregnancies among women. It is predicted that the Malay total fertility rates will likely fall to below the replacement level reaching 1.71 in 2040 whereas Chinese and Indian total fertility rates will substantially decrease to the lowest level in history below 1.0 which are 0.54 and 0.70 respectively. The evolution in Malaysian fertility rates is an alarming fact as, together with low mortality rates, it may impact the Malaysian population structure in future. Proactive policy measures are urgently needed to address these demographic shifts.

Numerical Analysis of Error From Sampling of Alternatives in Logit-Based Demand Forecasting Models with Massive Choice Sets

Use of the multinomial logit (MNL) functional form is widespread in transportation and land use demand modeling. As the state-of-the-art evolves away from aggregate forecasting and toward activity-based models and microsimulation, MNL models are being asked to accommodate increasingly large choice sets. Different strategies exist to address the challenge of estimating models with very large choice sets, but perhaps none is as commonly employed as sampling of alternatives. The effect of sampling of alternatives on parameter estimation has received considerable attention in the scientific literature. Yet comparatively little quantitative research exists that examines the issues that arise when the same sampling strategies are applied in the context of forecasting. In this study we conducted an analysis of the effect of sampling of alternatives in discrete choice models for disaggregate location choice forecasting. First, a novel measure of forecast error was defined and then used to quantify the extent of the problem as a function of sample rate, model error, and size of the universe of alternatives. Finally, we explored the potential for strategic sampling techniques to reduce the error observed under simple random sampling. In general, we found that the proportion of aggregate demand that was misallocated owing to sampling of alternatives was actually reduced as the size of the universe of alternatives increased. Additionally, simple random sampling was shown to outperform importance sampling in most scenarios. These findings suggest that error resulting from random sampling of alternatives is less of a problem for logit-based microsimulation models than previously assumed.

The Efficiency of Solid Waste Logistics Management: A Prominent Case Study of Banped Subdistrict, Municipality Mueang Khon Kaen District, Khon Kaen Province

Solid waste is a significant issue worldwide, and it is concerned by many countries worldwide. Local government has a substantial role in managing solid waste. Solid waste is increasing yearly because of several causes, such as the rapidly rising population and the massive city's expansion into suburban areas. The Municipal takes responsibility to collect and dispose of the solid waste in their responsible area. This article aims to forecast solid waste in the future using four methods: static methods, exponential smoothing, Holt's model, and Winter's model. Then, compare all methods to find the optimal ones with the most accurate forecasting. By determining the criteria, the forecast error measures are the absolute deviation (MAD) and the mean of the absolute percentage error (MAPE), which have a minimum error. One of the other purposes is to develop an optimization route that provides the possibility of decreasing distances and reducing transportation expenses. Typically, the most common methods are the Traveling Salesman Problem (TSP) and Vehicle Routing Problem (VRP) to solve the problems. This article will use two algorithms for two steps. The first step is to set the routes with the nearest neighbor heuristic (NN) to generate the routes MSW from point to point until all are reached and return to the origin point. In the second step, improve the routes by using the 2-Opt heuristic. The results found that the 2-Opt can decrease transportation distances, which is optimal and saves costs. In addition to that, it can make process management more effective. Keywords: Forecasting, Nearest Neighbour Heuristic, 2-Opt Heuristic, Travelling Salesman Problem

Comparison of ARIMA and Exponential Smoothing Models in Prediction of Stock Prices

Stock prices tend to show trends or seasonality or have random walk movements. Time series statistical models developed over time aid prediction of stock prices to assist informed decision-making for investors. These models provide quantitative information to financial specialists at the time of placing their buy–sell orders. The paper compares the movement of two univariate time series using two forecasting models—exponential smoothing and autoregressive integrated moving average (ARIMA) (p; d; q). We predict stock prices of selected 15 companies across three sectors (banking, pharmaceuticals, and Information technology) from NIFTY 50 data for the period April 01, 2016 to March 31, 2021. All these 15 companies are representative constituents of the three sectors within the Nifty 50 index. Performances of models were assessed through forecasting error measures such as root mean square error and mean absolute percentage error. Performances of both models were identical for nine stocks. Prediction based on ARIMA was more accurate for six stocks, whereas exponential smoothing model was a better indicator of stock prices in the case of one stock. However, the differences in error measures of the both the models are marginal, and parsimony principle may drive the choice of model.

Forecasting Total Fertility Rates For Urban And Rural Areas In Pakistan With A Coherent Functional Model.

To project total fertility rates for urban and rural areas in Pakistan up to 2027 with 80% prediction intervals. The secondary-data study was conducted from March 2020 to August 2020 at Data Bank in the Department of Statistics, University of Karachi, Pakistan, and comprised publicly available data of the Pakistan Demographic Survey from 1984 to 2007. Two statistical models, the functional time series model and the coherent functional model, were used to make the predictions about age-specific fertility rates. The forecasting performance of the models was compared through error measures. Data was analysed using R version 3.6.3. The predicted total fertility rate was 1.7 (80% prediction interval: 0.4-4.4) births per woman for urban areas and 2.2 (80% prediction interval: 0.6-5.3) births per woman for rural areas in 2027 using coherent functional model. The total fertility rate predicted by the functional time series model was 2.1 (80% prediction interval: 1.6- 2.6) births in urban areas and 2.7 (80% prediction interval: 1.7-4.1) births per woman in rural areas. The empirical comparison of forecast error measures obtained from the two models indicated that the coherent functional model performed better for forecasting total fertility rates for urban and rural areas in Pakistan. The projection of fertility rate obtained by the functional time series model and the coherent functional model described the future fertility behaviour of urban and rural populations.

Summaries of Articles

Summaries of Articles

Geographic prediction of crimes against property using Neural Networks and the SARIMA model

Predicting the number of crimes that will be committed in a certain geographical area is important for the management of resources destined for crime prevention. This research develops two predictive models of time series for the geographic prediction of property crimes in two districts of Chile Talcahuano and Hualpén. The models investigated were Neural Networks and SARIMA. Both models were trained and tested with the information provided by the Regional Prosecutor's Office of BioBío, Chile. The information contains the crimes against property committed in both districts between 2009 and 2019. The models were compared using the MAPE, MAD, and MSE forecast error measures. The comparison of the results does not show statistically significant differences between the results of both models.
 Keywords: Predictive Model, Time Series, Neural Networks, Crime Analitycs.

Short-Term Forecasting of Energy Production for a Photovoltaic System Using a NARX-CVM Hybrid Model

In this paper, a methodology for short-term forecasting of power generated by a photovoltaic module is reported. The method incorporates a nonlinear autoregressive with exogenous inputs (NARX) fed by the solar radiation and temperature times series, as well as an estimation of power time series obtained by implementing an ideal single diode model. This synthetic time series was validated against an actual photovoltaic module. The NARX model has been implemented in conjunction with the corrective vector multiplier (CVM) technique, which uses solar radiation under clear sky conditions to adjust the forecasting results. In addition, collinearity and the Granger causality tests were used to choose the input variables. The forecasting horizon was 24-h-ahead. The hybrid NARX-CVM model was compared to a nonlinear autoregressive neural network and persistence model using the typic forecasting error measures such as the mean bias error, mean squared error, root mean squared error and forecast skill. The results showed that the forecasting skills of the hybrid model are about 34% against the NAR model and about 42% against the Persistence model. The model was validated by blind forecasting. The results demonstrated evidence of the quality of the conformed forecasting model and the convenience of its implementation and building.

Real estate price estimation in French cities using geocoding and machine learning

This paper reviews real estate price estimation in France, a market that has received little attention. We compare seven popular machine learning techniques by proposing a different approach that quantifies the relevance of location features in real estate price estimation with high and fine levels of granularity. We take advantage of a newly available open dataset provided by the French government that contains 5 years of historical data of real estate transactions. At a high level of granularity, we obtain important differences regarding the models’ prediction powers between cities with medium and high standards of living (precision differences beyond 70% in some cases). At a low level of granularity, we use geocoding to add precise geographical location features to the machine learning algorithm inputs. We obtain important improvements regarding the models’ forecasting powers relative to models trained without these features (improvements beyond 50% for some forecasting error measures). Our results also reveal that neural networks and random forest techniques particularly outperform other methods when geocoding features are not accounted for, while random forest, adaboost and gradient boosting perform well when geocoding features are considered. For identifying opportunities in the real estate market through real estate price prediction, our results can be of particular interest. They can also serve as a basis for price assessment in revenue management for durable and non-replenishable products such as real estate.

Expert Systems in Intermittent Demand Forecasting

The main aim of the article was to propose an expert intermittent demand forecasting system. This system is based on stochastic simulation but also has many adjustments that might be treated as expert corrections. The presented forecasting system was applied in a company that sells mostly tools and work clothes. The number of forecasted items is over 16,000. The expert system was compared with the most popular intermittent demand forecasting methods (based on exponential smoothing) such as Croston’s, SBA, TSB or SES methods. For each of these methods a separate forecasting system was constructed. Smoothing factors as well as initial values for all forecasts were optimized. Forecasting accuracy measures taking into account the specificity of an intermittent demand time series were applied. Also, a new forecast error measure (scaled Absolute Mean Error) was proposed and used in the comparisons.

Patterns in Hotel Revenue Management Forecasting Systems: Improved Sample Sizes, Frozen Intervals, Horizon Lengths, and Accuracy Measures

Research in hotel revenue management system design has not paid much attention to the demand forecasting side of the system. And the research that has examined forecasting has tended to focus on the comparison of specific forecaster methodologies, as opposed to prioritizing how a total system should be parameterized: how far in the future should projections be, how much data to use to update each specific parameter, which measure of forecast error to use, and how long to freeze each parameter/forecast before updating. This paper fills this prioritization void by utilizing a full-functionality hotel reservation system simulation validated by the revenue management staff of a major hotel chain as the basis for running screening experiments on an exhaustive set of forecaster parameters with regards to their impact on bottom-line system performance (average nightly net revenue, where net revenue refers to total room rate receipts minus an overbooking per person penalty that estimates the discounted lost sales of future revenues). A screening experiment is run for each general type of operating environment (demand intensity, degree of market segment differentiation) that a property might face. We find that only two parameters are significant: the final combined forecast horizon length and how long that final forecast is frozen before updating. We find that these two factors interact in a negative fashion to influence net revenue.

Accuracy of Intermittent Demand Forecasting Systems in the Enterprise

Purpose: The main purpose of this article is to find the best forecasting method for intermittent demand times series, from the company’s point of view. Design/Methodology/Approach: Intermittent demand forecasting systems were constructed based on the Croston’s, SBA, TSB, SES and MA methods. A real database from the warehouse center, containing over sixteen thousand items, was used. Accuracy measures were also discussed. Forecasting methods were compared for all products and for separate demand categories (intermittent, lumpy, erratic, smooth). Findings: It was determined that the TSB method outperforms other methods for all products. The worst procedures were found to be Croston’s and SBA, which performed even worse than SES or MA. The same conclusions were true for intermittent and lumpy categories. In case of erratic and smooth items different results were obtained. It was determined that the SBA method performed best, while the TSB method yielded the poorest results. Practical Implications: The main conclusion is that to judge accuracy of forecasting systems first the proper forecast error measures should be chosen. Based on obtained results, TSB method seems to be the best for intermittent demand times series and this method is recommended for enterprises dealing with intermittent demand. Originality/value: Since such error measures as MASE or scaled MAE favored an underestimated (or even zero) forecast, in the article a new error metric is proposed, which was named scaled Compound Error (sCE). It is a scaled error, and it considers forecast biasedness.

Performance Assessment of Time Series Forecasting Models for Cloud Datacenter Networks’ Workload Prediction

The resource scaling has been influential in enabling the cloud service providers to provision the resources on-demand effectively. The prior estimation of workloads helps in addressing the scaling issues arises due to dynamic nature of the resource demands. In this paper, we evaluate six different forecasting approaches over real world workload data traces of web and cloud servers. The entire analysis is carried out three times as three different functions are used to measure the deviation in forecasts. The three forecast error measures are root mean squared error, mean absolute error, and mean absolute scaled error. We also carried out a statistical evaluation using Friedman test and Finner post-hoc analysis. The study concludes that the auto ARIMA process outperforms other models and achieves the best rank in the statistical analysis.

Incorporating crossed classification credibility into the Lee–Carter model for multi-population mortality data

Recent developments in actuarial literature have shown that credibility theory can serve as an effective tool in mortality modelling, leading to accurate forecasts when applied to single or multi-population datasets. This paper presents a crossed classification credibility formulation of the Lee–Carter method particularly designed for multi-population mortality modelling. Differently from the standard Lee–Carter methodology, where the time index is assumed to follow an appropriate time series process, herein, future mortality dynamics are estimated under a crossed classification credibility framework, which models the interactions between various risk factors (e.g. genders, countries). The forecasting performances between the proposed model, the original Lee–Carter model and two multi-population Lee–Carter extensions are compared for both genders of multiple countries. Numerical results indicate that the proposed model produces more accurate forecasts than the Lee–Carter type models, as evaluated by the mean absolute percentage forecast error measure. Applications with life insurance and annuity products are also provided and a stochastic version of the proposed model is presented.

Assessing the impact of observations in a multi‐year reanalysis

AbstractOperational and quasi‐operational weather prediction centres have been routinely assessing the contribution from various observing systems to reducing errors in short‐range forecasts for a number of years now. The original technique, Forecast Sensitivity‐based Observation Impact (FSOI), involves definition of a forecast error measure and evaluation of sensitivities with respect to changes in the observations that require adjoint operators of both the underlying tangent linear model and corresponding analysis technique. The present work applies FSOI to reanalysis and aims at providing an expanded view of the contribution of various observing systems over nearly 40 years of assimilation. Specifically, this study uses MERRA‐2, given that its supporting software includes all ingredients necessary to calculate FSOI. Part of this work shows how the quality of forecasts improves over the course of the reanalysis, and examines forecast sensitivities relevant to FSOI. The assessment here finds that, for example, conventional observations are a major player in reducing forecast error throughout the 40 years of reanalysis, even when their volume reduces from 45% in the earlier periods to about 5% in the modern era; satellite radiances, especially microwave instruments are major contributors to error reduction from the early single platform TIROS‐N days to the current multi‐platform scenario, though their fractional contribution reduces slightly from the early 2000's onward after the increased availability of wind observation from aircraft and atmospheric motion vectors, and the introduction of GPSRO; infrared instruments play a secondary role to microwave but are significant still, with the peculiar result of fractional impacts contribution from modern hyperspectral instruments being roughly similar to those from early infrared instruments. The dependence of results on the chosen error measure is emphasized throughout.

Two-step-hybrid model based on data preprocessing and intelligent optimization algorithms (CS and GWO) for NO2 and SO2 forecasting

The harm of air pollution to human health has received close attention, and many scholars have done research on the relationship between air pollution and human health. NO2 and SO2 are two important air pollutants, which are the two major sources for acid rain. Acid rain has potential harm to water, soil, plants, buildings and people’s health. So, it is significant to develop effective model for NO2 and SO2 forecasting and warning. However, it is not easy to get precise forecasting results for the irregular NO2 and SO2 series. This research focuses on modeling and prediction of the two major sources of acid rain, NO2 and SO2, and four cities in Central China region are selected as the test data. Specially, Central China region has the most severe regions with the acid rain in China for a long time. The proposed procedure is named as two-step-hybrid model, the detailed procedure of the proposed model can be summarized as three steps: First, the original NO2 (or SO2) sequence is decomposed into high-frequency and low-frequency sequences by the Complementary Ensemble Empirical Mode Decomposition (CEEMD); Second, Support Vector Regression (SVR) model combined the Cuckoo Search algorithm (CS) and Grey Wolf Optimizer algorithm (GWO) are employed to model the high-frequency and low frequency sequences, respectively; Third, forecasting data of low frequency and high frequency are summed as the final prediction results for NO2 (or SO2). In terms of model selection and assessment process, the proposed model and the other established models are compared by the forecasting error measures, such as MAE, MAPE and RMSE. The forecasting comparisons showed that the proposed two-step-hybrid model based on CEEMD, SVR, CS and GWO has higher forecasting precision compared with other forecasting models. Specially, the hybrid model CEEMD-CS-GWO-SVR, the low-frequency data using the SVR-CS and the high frequency data using SVR-GWO, is the best model for the prediction of NO2 and SO2 for the cities in Central China.

New Forecasting Technique for Intermittent Demand, Based on Stochastic Simulation. An Alternative to Croston’s Method

The main aim of the article is to present a new forecasting technique, applicable in case of intermittent demand. To present properties of this new technique, the accuracy of the predictions generated by the Croston’s method and by the author’s method (based on stochastic simulation) was analyzed. For comparison, methods such as moving average and simple exponential smoothing are as well used as a reference. Also the SBA method, a modification of Croston’s method, is applied. Croston’s method is an extension of adaptive methods. It separates the interval between the (non‑zero) sales and the sales level. Its purpose is to better forecast intermittent (sporadic) demand. The second prognostic method is the author’s proposal which relies on two stages. In the first stage, based on stochastic simulation, it determines if an event (sale) occurs in a given period. In the second stage, the sales level is estimated (if the previous stage shows that the sales will occur). Due to the strong asymmetry of the sales, the sales level is determined on the basis of the corresponding quantiles. The basis for forecasting are weekly sales series of about fourteen thousand products (real data). The analyzed time series can be defined as atypical, which is manifested by a small number of non‑zero observations (high number of zeros), high volatility and randomness (randomness tests indicate white noise). Forecast error measures are used to characterize both the bias and the efficiency. The forecast error measures will be characterized so that they can be applied to a time series with a large number of zeros (including the author’s forecast error measure proposal). Forecasts were evaluated with respect to the distributions of four ex post errors, such as mean error (ME), mean absolute deviation (MAD), mean absolute scaled error (MASE) and the author’s proposal (error D). The proposed technique, based on stochastic simulation, seems to be the least biased and most efficient. The Croston’s method gives positively biased predictions with rather low efficiency. The proposed forecasting technique might support decisions in enterprises facing the problem of forecasting intermittent demand. The more accurate forecasts could increase the quality of customer service and optimize the inventory level.

Quantile range-based volatility measure for modelling and forecasting volatility using high frequency data

Volatility of asset prices in financial market is not directly observable. Return-based models have been proposed to estimate the volatility using daily closing prices. Recently, many new range-based volatility measures were proposed to estimate the financial volatility. This paper proposes quantile Parkinson (QPK) measure to estimate daily volatility and to show how it can robustify the Parkinson (PK) measure in the presence of intraday extreme returns. Results from extensive simulation studies show that the QPK measure is more efficient than intraday (open-to-close) squared returns and PK measures in the presence of intraday extreme returns. To demonstrate the applicability of QPK measure, we analyse the daily Standard and Poor 500 index by fitting the QPK measure to the conditional autoregressive range (CARR) models. Results show that choosing a suitable interquantile level width for the QPK measure will reduce its variance and hence improve its efficiency. In addition, the QPK measure using asymmetric CARR model gives the best in-sample model fit based on Akaike information criterion and provides the best out-of-sample forecast based on root mean squared forecast error and other measures. Mincer Zarnowitz test is carried out to confirm the unbiasedness of the forecasted volatility. Different levels of value-at-risk and conditional value-at-risk forecasts are also provided and tested.

Evaluation of forecasting methods in aggregate production planning: A Cumulative Absolute Forecast Error (CAFE)

Abstract The purpose of forecasting error measures is to estimate forecasting methods and choose the best one. Most typical forecasting error measures are designed based on the gap between forecasts and actual demands and, consequently, a forecasting method yielding forecasts in accordance with real demands is considered as good. However, in some applications such as aggregate production planning, these measures are not suitable because they are not capable for considering any effects caused by forecasting error such as increasing cost or decreasing profit. To tackle this issue, we propose a new measure, CAFE (Cumulative Absolute Forecast Error), to evaluate forecasting methods in terms of total cost. Basically, the CAFE is designed to consider not only forecasting errors but also costs occured by errors in aggregate production planning which is set up based on forecasts. The CAFE is a product sum of cumulative forecasting error and weight factors for backorder and inventory costs. We have demonstrated the effectiveness of the proposed measure by conducting intensive experiments with demand data sets from M3-competition.