Theory Testing in a World of Constrained Research Design

Abstract

Because censored sampling is often unavoidable in much sociological data analysis, computationally simple corrections of censoring bias would be useful. Heckman's correction is simple to compute, widely used, and proven asymptotically correct under certain assumptions, but its limitations in practical situations are not well known in sociology. Here, we overview prior criticisms of Heckman's estimator, and we consider the case in which its normality assumptions are satisfied, censoring rates are high, and sample sizes are small. Results of 14,400 analyses of computer-generated simulation data suggest that Heckman's method performs well under certain circumstances, but that it very frequently worsens estimates, especially under conditions that are likely to be present in sociological data. Thus, the technique is probably not a general cure for censoring bias in sociology, except perhaps where strong theory permits certain strong assumptions. We reconsider censored sampling correction strategies in the context of statistical analysis as a theory-building tool, with emphasis on research strategy in the presence of irremediable censoring bias.