Quality Review of Mass Adjudication: A Randomized Natural Experiment at the Board of Veterans Appeals, 2003–16

Abstract

We study a unique natural experiment, during which 5–10% of draft opinions by judges of the Board of Veterans Appeals (BVA) were randomly selected for “quality review (QR)” by a team of full-time staff attorneys. The express goals of this performance program were to measure accuracy and reduce remands on appeal. In cases of legal error, the QR team wrote memoranda to judges for correction of draft opinions. We use rich internal administrative data on nearly 600,000 cases from 2003 to 2016 to conduct the first rigorous evaluation of this program. With precise estimates, we show that QR had no appreciable effects on appeals or remands. Based on internal records, we demonstrate that this inefficacy is likely by design, as meeting the performance measure of “accuracy” conflicted with error correction. These findings inform longstanding questions of law, organization, and bureaucracy, including performance management, standards of review, and institutional design of adjudication.