Benchmarks for Discrete Choice Models

Abstract

In assessing the performance of a choice model, we have to answer the question, “Compared with what?” Analyses of consumer brand choice data historically have measured fit by comparing a model's performance with that of a naive model that assumes a household's choice probability on each occasion equals the aggregate market share of each brand. The authors suggest that this benchmark could form an overly naive point of reference in assessing the fit of a choice model calibrated on scanner-panel data, or any repeated-measures analysis of choice. They propose that fairer benchmarks for discrete choice models in marketing should incorporate heterogeneity in consumer choice probabilities, evidence for which is by now well documented in the marketing literature. They use simulated data to compare the performance of parametric and nonparametric benchmark models, which allow for heterogeneity in consumer choice probabilities, with the performance of the aggregate share-based benchmark model, which assumes consumers are homogeneous in their choice probabilities. They also assess the performance of two previously published consumer behavior models against the proposed fairer benchmark models that allow for heterogeneity in consumer choice probabilities. They find that one provides a significantly better fit than their more conservative benchmark models and the other performs less favorably.