Linear Probability Models (LPM) and Big Data: The Good, the Bad, and the Ugly

Abstract

Sizes of datasets used in academic research are growing quickly, with many studies using tens and hundreds of thousands or even millions of records. Linear regression is among the most popular statistical model in social sciences research. Linear probability models, which are linear regression models applied to a binary outcome, are commonly used for various reasons, despite criticisms of such usage. We carry out an extensive study to evaluate the use of LPMs in the realm of Big Data, where large samples and many variables are available. We evaluate performance in terms of coefficient estimation as well as predictive power. We compare performance to alternatives suggested in the literature. We find that the LPM is beneficial for descriptive modeling when the outcome is naturally binary, whereas it is beneficial for predictive modeling when the outcome is binary by discretization. We motivate and illustrate our study through an application to modeling price in online auctions, using real data from the online auction site eBay.