Empirically Validating Citation Metrics for Legal Scholars: A Market Approach

Abstract

Citation counts are a common quantitative metric used by researchers and analysts to assess scholarly output. When U.S. News & World Report announced in 2019 that it was developing a citation-based ranking for law schools, it brought new attention to debates about citations counts in legal scholarship. Supporters of citation metrics argue that they are superior to alternative measures of scholarly reputation, such as surveys. Critics are skeptical that citations serve as a meaningful proxy for scholarly quality and raise concerns that citation metrics could distort the incentives of law professors and faculties. We examine the validity of citation metrics by examining how well they correspond to a “market valuation” of legal scholars in lateral hiring. We consider two outcomes: whether professors make lateral moves and the rank of the institution where they are hired. Using citation counts derived from the HeinOnline database, we find that citation metrics have a weak association with lateral outcomes. Metrics that mitigate the effect of the highly skewed distribution of citations, such as log citations and the h-index, perform slightly better. Article placements are stronger predictors of lateral outcomes. In particular, articles in top law reviews, top peer-reviewed journals, and online law reviews are all associated with moving to higher-ranked faculties, even after controlling for citations. This implies that citation rankings undervalue these kinds of publications. The divergence between citation counts and professors’ market valuation suggest that citation rankings could significantly distort publication and hiring in the legal academy.