The efficacy of measuring judicial ideal points: The mis-analogy of IRTs

Abstract

IRT models are among the most commonly used latent trait models in all of political science, particularly in the estimation of ideal points of political actors in institutions. While widely used, IRT models are often misapplied, and a key element of their estimation, the item parameters, are almost always ignored and discarded. In this paper, we look into the application of IRT models to the estimation of judicial ideology scores by Martin and Quinn (2002). Building off of a replication and extension of Martin and Quinn (2002), we demonstrate that the often-ignored item parameters are, in fact, inconsistent with the assumptions of IRTs. Then, using a post-estimation fix that is designed to ameliorate the problem, we run the model again, generating new scores. We then compare our new ideal points to the existing ideal points and discuss the implications for both ideal point modeling generally and in judicial politics specifically. We conclude by replicating a prominent study in judicial politics that demonstrates how inconsistencies in the estimation of IRT models can be consequential and bring up concerns with the implications for what this could mean for the usefulness of scores estimated via IRT models.