The Big Data Newsvendor: Practical Insights from Machine Learning

Abstract

We investigate the data-driven newsvendor problem when one has n observations of p features related to the demand as well as historical demand data. We propose two approaches to finding the optimal order quantity in this new setting -- Machine Learning (ML) with and without regularization, and Kernel-weights Optimization (KO). We show that the resulting Big Data newsvendor problem can be solved by LP, MIP or QCQP programs under the ML approach, and by a simple sorting algorithm under the KO approach. We justify the use of feature information by showing that not including them yields inconsistent decisions, which translates to sub-optimal costs even with infinite amount of demand data. We then derive finite-sample performance bounds on the out-of-sample costs of the feature-based decisions, which shows (i) the Big Data regime, when over-fitting dominates finite-sample bias, is defined by p > O(n^{-1/(2 8/p)}\sqrt{\log{(n)}}), and (ii) both regularized ML and KO are effective methods to handle over-fitting. Finally, we apply the feature-based algorithms for nurse staffing in a hospital emergency room using a data set from a large UK teaching hospital and find that (i) the best KO and ML algorithms beat the best practice benchmark by 23% and 24% respectively in the out-of-sample cost with statistical significance at the 5% level, and (ii) the best KO algorithm is faster than the best ML algorithm by three orders of magnitude and the best practice benchmark by two orders of magnitude.