The Perceived Unreliability of Rank-Ordered Data: An Econometric Origin and Implications

Abstract

The problem of unstable coefficients in the rank-ordered logit model has been traditionally interpreted as a sign that survey respondents fail to provide reliable ranking responses. This paper shows that the problem may embody the inherent sensitivity of the model to stochastic misspecification instead. Even a minor departure from the postulated random utility function can induce the problem, for instance when rank-ordered logit is estimated whereas the true additive disturbance is iid normal over alternatives. Related implications for substantive analyses and further modelling are explored. In general, a well-specified random coefficient rank-ordered logit model can mitigate, though not eliminate, the problem and produce analytically useful results. The model can also be generalised to be more suitable for forecasting purposes, by accommodating that stochastic misspecification matters less for individuals with more deterministic preferences. An empirical analysis using an Australian nursing job preferences survey shows that the estimates behave in accordance with these implications.