Systematic Content Analysis of Judicial Opinions

Abstract

Despite the interdisciplinary bent of legal scholars, the academy has yet to identify an empirical methodology that is uniquely its own. We propose that one standard social science technique - content analysis - could form the basis for an empirical methodology that is uniquely legal. It holds the potential for bringing social science rigor to our empirical understanding of caselaw, and therefore for creating what is distinctively a legal form of empiricism. To explore this potential, we collected all 122 examples we could find that use content analysis to study judicial opinions, and coded them for pertinent features. Legal scholars began to code and count cases decades ago, but use of this method did not accelerate until about 15 years ago. Most applications are home-grown, with no effort to draw on established social science techniques. To provide methodological guidance, we survey the questions that legal scholars have tried to answer through content analysis, and use that experience to generalize about the strengths and weaknesses of the technique compared with conventional interpretive legal methods. The epistemological roots of content analysis lie in legal realism. Any question that a lawyer might ask about what courts say or do can be studied more objectively using one of the four distinct components of content analysis: 1) replicable selection of cases; 2) objective coding of cases; 3) counting case contents for descriptive purposes; or 4) statistical analysis of case coding. Each of these components contributes something of unique epistemological value to legal research, yet at each of these four stages, some legal scholars have objected to the technique. The most effective response is to recognize that content analysis does not occupy the same epistemological ground as conventional legal scholarship. Instead, each method renders different kinds of insights that complement each other, so that, together, the two approaches to understanding caselaw are more powerful that either alone. Content analysis is best used when each decision should receive equal weight, that is, when it is appropriate to regard the content of opinions as generic data. Scholars have found that it is especially useful in studies that question or debunk conventional legal wisdom. Content analysis also holds promise in the study of the connections between judicial opinions and other parts of the social, political, or economic landscape. The strongest application is when the subject of study is simply the behavior of judges in writing opinions or deciding cases. Then, content analysis combines the analytical skills of the lawyer with the power of science that comes from articulated and replicable methods. However, analyzing the cause-and-effect relationship between the outcome of cases and the legally relevant factors presented by judges to justify their decisions raises a serious circularity problem. Therefore, content analysis is not an especially good tool for helping lawyers to predict the outcome of cases based on real-world facts. This article also provides guidance on the best practices for using this research method. We identify techniques that meet standards of social science rigor and account for the practical needs of legal researchers. These techniques include methods for case sampling, coder training, reliability testing, and statistical analysis. It is not necessary to practice this method profitably only at its highest level. Instead, we show that valuable uses can be made even by those who are largely innumerate.