Sample Selection Bias in Measuring Online Advertising Effectiveness

Abstract

A goal of measuring the effectiveness of online advertising is to predict the impact of advertising on the users perception of the advertised brand. Historically these models have been parameterized using survey data only and the bias inherent in such surveys is discussed in isolation. In this paper we examine the impact of selection/survey bias on the estimation of Ad effectiveness for two products using a censored bivariate Probit model. We find that survey completion reveals substantive differences in brand response. Put another way, a significant correlation exists between the unobserved heterogeneity in the survey model and the error term in response model. For one product, we conclude that factors that make it more likely to complete the survey make it less likely to recall the ad. Consideration of selection bias has the following impacts: standard deviation of predicted outcome widens to reflect a more diverse/general population; the impact of variables such as product ownership and gender are reduced; and, most importantly, recall of general population is estimated to be greater than measured recall of self selected participants. For the other product, we conclude that factors that make it more likely to complete the survey make it more likely to consider purchasing the product. Consideration of selection bias has the following impacts: the observed purchase consideration on self selected sample overestimates the population purchase consideration; and while the treatment is effective in influencing purchase consideration among general population its impact is less than the measured impact on self selected population.