Forecast UPC-level FMCG demand, Part I: Exploratory analysis and visualization

Abstract

We are interested in forecasting a large collection of FMCG demand time series. As the demand of FMCG exists in a hierarchy (from manufacturers to distributors to retailers), the bottom level of the hierarchy may contain thousands or even millions of time series. Producing aggregate consistent forecasts while utilizing the unique features from each time series thus become a technical challenge. To achieve better forecasting results, exploratory analysis is often necessary to obtain insights on the underlying demand generating mechanism for each time series. Exploratory analysis aims at discovering those so-called factors, such as price, demand of the complementary/substitutive goods and calendar events, which can help explain some of the demand fluctuation. During forecast accuracy evaluation, outlier detection is also important; a single anomalous time series can contribute much to the overall error. However, in a big data (such as retailing scanner data) enabled environment, exploratory analysis and visualization need much attention, because of the non-scalable nature of the existing methods. Scalability is essential for exogenous factor selection and outlier detection in big time series data. In Part I of this two-part paper, we introduce some exploratory analytics and visualization methods (from not scalable to very scalable) for big retailing time series. Forecasting of the hierarchical FMCG demand is addressed in Part II.