Logistic Regression Bias Correction for Large Scale Data with Rare Events

Abstract

Logistic regression is a classical classification method, it has been used widely in many applications which have binary dependent variable. However, when the data sets are imbalanced, the probability of rare event is underestimated in the use of traditional logistic regression. With data explosion in recent years, some researchers propose large scale logistic regression which still fails to consider the rare event, therefore, there exists bias when applying their models for large scale data sets with rare events. To address the problems, this paper proposes LRBC method to correct bias of logistic regression for large scale data sets with rare events. Empirical studies compare LRBC with several state-of-the-art algorithms on an actual ad clicking data set. It demonstrates that LRBC method is able to exhibit much better classification performance, and the distributed process for bias correction also scales well.