Is the Effect Larger in Group A or B? It Depends: Understanding Results From Nonlinear Probability Models.

Abstract

Demographers and other social scientists often study effect heterogeneity (defined here as differences in outcome-predictor associations across groups defined by the values of a third variable) to understand how inequalities evolve between groups or how groups differentially benefit from treatments. Yet answering the question "Is the effect larger in group A or group B?" is surprisingly difficult. In fact, the answer sometimes reverses across scales. For example, researchers might conclude that the effect of education on mortality is larger among women than among men if they quantify education's effect on an odds-ratio scale, but their conclusion might flip (to indicate a larger effect among men) if they instead quantify education's effect on a percentage-point scale. We illuminate this flipped-signs phenomenon in the context of nonlinear probability models, which were used in about one third of articles published in Demography in 2018-2019. Although methodologists are aware that flipped signs can occur, applied researchers have not integrated this insight into their work. We provide formal inequalities that researchers can use to easily determine if flipped signs are a problem in their own applications. We also share practical tips to help researchers handle flipped signs and, thus, generate clear and substantively correct descriptions of effect heterogeneity. Our findings advance researchers' ability to accurately characterize population variation.