A Framework for Credibility in Empirical Legal Analysis

Abstract

Quantitative empirical research is central to both legal scholarship and litigation, but there is little confidence in it. This is because researchers can manipulate data to arrive at any result they seek to find. The root of the problem is data fishing, the practice of using data to search for and selectively report results that are statistically significant or otherwise favorable to the researcher. For reasons explained in this article, data fishing invalidates statistical results by causing false positives and false impressions. It creates an environment in which, at best, readers are highly skeptical of statistical analysis and, at worst, they base important decisions, such as policy decisions and jury verdicts, on incorrect information. The practice is nevertheless prevalent in law—often committed by well-intentioned researchers who are unaware of its harms or unaware that their analysis constitutes data fishing. This article exposes the harm that data fishing in empirical legal research causes. It then develops a framework for eliminating data fishing and restoring confidence in empirical analysis in legal scholarship and litigation. This framework, which I call DASS (an acronym for Design, Analyze, Scrutinize, and Substantiate), builds on methods in statistics and is designed for researchers to use to safeguard against data fishing and for consumers of empirical research—including scholars, courts, policymakers, and members of the public—to use to evaluate the reliability of a researcher’s statistical claims. DASS is designed to be simple and flexible, tailored to suit empirical research in law, and a substantial advancement over current anti-data-fishing practices in the social sciences, which have generally been ineffective. It can be applied broadly as a framework for credibility in empirical legal research, as well as to address a range of classical challenges in litigation, such as the hired-gun and battle-of-the-experts problems in evidence law.