The Problem of Data Bias in the Pool of Published U.S. Appellate Court Opinions

Abstract

For decades, researchers have studied the relationship between the political leanings of judges and the outcomes of appellate litigation in the United States. The primary source of data for this research has been published judicial opinions that describe cases and their outcomes. However, only a relatively small number of cases result in published opinions, and this sample of cases may be subject to serious biases. Based on computational text analysis of over 150,000 published opinions issued by federal appellate courts in the years 1970–2010, we find strong evidence of data bias based on relationships between the party affiliations of judges on appellate court panels and the characteristics of cases that result in published opinions. These relationships imply that the inferential model that underlies much of the judicial politics literature can lead to biased or spurious findings concerning the causal influence of judicial attributes on case outcomes.