On the prediction of rare events when sampling from large data

Abstract

When modeling rare events using logistic regression, independent samples of event occurrence (ones) and nonoccurrence (zeros) are commonly taken from large datasets in order to fit models efficiently. A deterministic offset may then be included in the model to compensate for that sampling method. We propose a more complex sampling approach using stratified sampling within the sets of ones and zeros to ensure that we may sample more zeros from strata of interest. This design may avoid situations in which a random sample of zeros fails to capture the range of a key covariate. We employ sampling weights along with stratum-specific intercepts to obtain unbiased estimates of the logistic regression coefficients (including the intercept) and their standard errors. We use simulation to show that this method provides unbiased parameter estimates comparable with those of maximum likelihood. We also illustrate an application of this method to wildland fire occurrence prediction in a study area in northwestern Ontario, Canada.