Data pruning in consumer choice models

Abstract

Common, if not ubiquitous, Marketing practice when estimating models for scanner panel data is to: (a) observe the data, (b) prune the data to a “manageable” number of brands or SKUs, and (c) fit models to the remaining data. We demonstrate that such pruning practice can lead to significantly different (and potentially biased) elasticities, and hence different managerial/practical outcomes, especially in the context of model misspecification. We first justify our claims theoretically by writing the general problem in a classic missing-data framework and demonstrate that commonly used pruning mechanisms (gleaned from current academic Marketing literature) can lead to a nonignorable missing data mechanism. Secondly, we summarize an extensive set of simulations that were run to understand the driving factors of that bias. The results indicate much greater pruning bias in those cases where model fit is poor (small \(R^{2}\)), random utility errors are correlated with the covariates, or the model is misspecified (e.g., a homogeneous logit is specified when a mixed-logit is true). Empirically, we also demonstrate our findings on the well-cited and highly utilized fabric softener data of Fader and Hardie (1996). Our empirical findings suggest a number of estimates that vary according to the way in which the data is pruned including the magnitude of market mix and attribute elasticities, and purchase probabilities, but that the pruning effect is smaller for better fitting models.