Consequences of Ignoring Ideal Point Items for Applied Decisions and Criterion-Related Validity Estimates

Abstract

Although the popularity of ideal point measurement continues to grow, the implications of mis-scoring ideal point scales by computing a sum-score have received little empirical attention. The possibility of incorrectly scoring these scales is quite high given that many scales developed under dominance assumptions have been shown to follow ideal point assumptions. Unfortunately, a systematic investigation of these implications is missing; this study looked to fill this gap. A Monte Carlo simulation was conducted to determine the impact of sum-scoring an ideal point scale on the rank ordering of simulees, bias in criterion-related validity estimates, and selection outcomes. The simulation varied the number of meaningfully unfolding items and the true criterion-related validities. Analyses were conducted within the full sample and varying selection ratios. With all simulees, and when 20 % or more of the items meaningfully unfolded, rank ordering of simulees changed, criterion-related validity estimates showed downward bias, selection hit rates reduced, and per-hire utility was reduced. The patterns of results were similar, though of stronger magnitude, as the sample analyzed became more restrictive. When respondents follow the ideal point response process and the scale contains even a small portion of meaningfully neutral items, sum-scores are inappropriate indices of attribute standing. The consequences of ignoring meaningfully neutral items were much worse in typical application situations. Although some studies have shown strong correlations among the sum-score and IRT-based attribute estimates for ideal point scales, other studies have shown attenuate relations. We present the first systematic investigation of what has, heretofore, been demonstrated incidentally.